
mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


Sustained 

Progress 

New Findings About the Effectiveness and 
Operation of Small Public High Schools of 
Choice in New York City 


Howard S. Bloom 
Rebecca Unterman 


AUGUST 2013 



Sustained Progress 


New Findings About the Effectiveness and Operation of 
Small Public High Schools of Choice in New York City 


Howard S. Bloom 
Rebecca Unterman 


August 2013 


mdrc 

BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


Funding for this report was provided by the Bill & Melinda Gates Foundation. 

Dissemination of MDRC publications is supported by the following funders that help finance 
MDRC’s public policy outreach and expanding efforts to communicate the results and 
implications of our work to policymakers, practitioners, and others: The Annie E. Casey 
Foundation, The George Gund Foundation, Sandler Foundation, and The Starr Foundation. 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. 
Contributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell 
Foundation, An heuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart 
Mott Foundation, Ford Foundation, The George Gund Foundation, The Grable Foundation, The 
Lizabeth and Frank Newman Charitable Foundation, The New York Times Company 
Foundation, Jan Nicholson, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler 
Foundation, and The Stupski Family Fund, as well as other individual contributors. 

The findings and conclusions in this report do not necessarily represent the official positions or 
policies of the funders. 


For information about MDRC and copies of our publications, see our Web site: www.mdrc.org. 
Copyright © 2013 by MDRC®. All rights reserved. 



Overview 


In 2002, New York City embarked on an ambitious and wide-ranging series of education 
refomis. At the heart of its high school refonns were three interrelated changes: the institution 
of a districtwide high school choice process for all rising ninth-graders, the closure of 3 1 large, 
failing high schools with an average graduation rate of 40 percent, and the opening of more than 
200 new small high schools. 

Over half of the new small schools created between the fall of 2002 and the fall of 2008 
were intended to serve students in some of the district’s most disadvantaged communities and 
are located mainly in neighborhoods where large, failing high schools had been closed. MDRC 
has previously released two reports on these “small schools of choice,” or SSCs (so called 
because they are small, are academically nonselective, and were created to provide a realistic 
choice for students with widely varying academic backgrounds). Those reports found marked 
increases in progress toward graduation and in graduation rates for the cohorts of students who 
entered SSCs in the falls of 2005 and 2006. The second report also found that the increase in 
graduation rates applied to every student subgroup examined, and that SSC graduation effects 
were sustained even after five years from the time sample members entered high school. 

This report updates those previous findings with results from a third cohort of students, 
those who entered ninth grade in the fall of 2007. In addition, for the first time it includes a look 
inside these schools through the eyes of principals and teachers, as reported in interviews and 
focus groups held at the 25 SSCs with the strongest evidence of effectiveness. 

In brief, the report’s findings are: 

• SSCs in New York City continue to markedly increase high school 
graduation rates for large numbers of disadvantaged students of color, even 
as graduation rates are rising at the schools with which SSCs are compared. 

• The best evidence that exists indicates that SSCs may increase graduation 
rates for two new subgroups for which findings were not previously 
available: special education students and English language learners. 
However, given the still-limited sample sizes for these subgroups, the 
evidence will not be definitive until more student cohorts can be added to the 
analysis. 

• Principals and teachers at the 25 SSCs with the strongest evidence of 
effectiveness strongly believe that academic rigor and personal relationships 
with students contribute to the effectiveness of their schools. They also 
believe that these attributes derive from their schools’ small organizational 
structures and from their committed, knowledgeable, hardworking, and 
adaptable teachers. 
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Preface 


Too many young people growing up in U.S. cities are not succeeding in public high schools: 
they graduate unprepared for postsecondary education and the workforce or do not graduate at 
all. While urban districts have tried an array of turnaround strategies, success on a large scale is 
rare, in part because budgets, political pressures, or leadership changes preclude innovations 
that address fundamental issues, prevent them from being tried for long enough to succeed, or 
keep them from expanding to reach enough students when they do succeed. That’s why the sto- 
ry of high school refonn in New York City is special. 

In 2002, New York City embarked on an ambitious set of refonns. It instituted a dis- 
trictwide high school choice process for all rising ninth-graders, closed large, failing high 
schools, and opened more than 200 new small high schools. More than half of the new schools 
created between 2002 and 2008 were intended to serve students in some of the district’s most 
disadvantaged communities. The schools were developed and approved through a competitive 
proposal process administered by the New York City Department of Education, which in early 
years was conducted in collaboration with the United Federation of Teachers and the Council of 
School Supervisors and Administrators, a consortium of funders, and, through the New Century 
High School Initiative, New Visions for Public Schools. The proposal process was designed to 
stimulate innovative ideas for new schools by a range of stakeholders and institutions, from ed- 
ucators to school reform intermediary organizations, such as New Visions for Public Schools, 
the Urban Assembly, the Institute for Student Achievement, and the College Board. This reform 
effort represents innovation on a large scale, sustained for over a decade so far. 

This report offers new — and very encouraging — findings from MDRC’s ongoing 
study of these small, academically nonselective schools, which MDRC calls Small Schools of 
Choice (SSCs) because they offer a real choice for students with varying academic back- 
grounds. Earlier reports (in 2010 and 2012) received wide attention for their findings that SSCs 
boosted graduation rates for a variety of disadvantaged students of color. This report, which in- 
cludes results for a new cohort of students, confirms that students at SSCs are more likely to 
graduate than students at other New York City schools, even as graduation rates at those other 
schools have risen. For the first time, this report also offers qualitative findings about what prin- 
cipals and teachers at the SSCs with the strongest evidence of effectiveness think are the reasons 
for their success: they overwhelmingly cite academic rigor and personal relationships with stu- 
dents. 


Why are these findings important? The nation’s attention is focused on turning around 
failing urban high schools, and this study provides convincing evidence that large-scale trans- 
fonnation is possible in a large, urban public school system. Serving low-income students of 
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color, two-thirds of whom were far behind grade level when they started the ninth grade, SSCs 
are improving the lives and life prospects of many young people. 

But while these results are historic, and hold great implications for refonning failing 
high schools in other communities, more remains to be done. More than a quarter of SSC stu- 
dents still fail to graduate high school within five years. Next year, MDRC will publish a report 
examining how impacts on graduation vary among SSCs, which should yield some lessons 
about what operational factors lead to better results for students. That infonnation should in turn 
begin to provide a blueprint for future reforms, not just in New York but across the nation. In 
the future, MDRC also plans to follow students to see whether SSCs contribute to success in 
postsecondary education and the labor market. 


Gordon L. Berlin 
President 
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Introduction 

In 2002, New York City embarked on an ambitious and wide-ranging series of education 
reforms. With the advent of mayoral control, the New York City Department of Education 
(NYCDOE) was centralized, and individual schools were granted greater autonomy over their 
curricula, professional development, and teacher hiring in exchange for strict accountability 
standards and public perfonnance assessments. 1 In addition these reforms created an array of 
interventions, including a system of transfer schools for students who were close to dropping 
out of high school, programs to produce new school leaders, and a (Fair Student FundingO 
model that provides resources to schools based in part on the educational needs of their stu- 
dents. 2 At the heart of the high school component of these efforts were three interrelated 
refomis: the institution of a districtwide high school choice process for all rising ninth-graders, 
the closure of 3 1 large, failing high schools with an average graduation rate of 40 percent, 3 and 
the opening of more than 200 new small high schools. 

Over half of the new small schools (123 of them) created between the fall of 2002 and 
the fall of 2008 were intended to serve students in some of the district® most disadvantaged 
communities and are located mainly in neighborhoods where large, failing high schools had 
been closed (see Figure l). 4 Because these schools are small, are academically nonselective, and 
were created to provide a realistic choice for students with widely varying academic back- 
grounds, MDRC researchers call them fkmall schools of choice, 6 or SSCs. 5 In addition to their 
uniformly small size and location mainly in disadvantaged communities, SSCs have other 
defining traits: 

• All were created through a competitive proposal process in which teachers, 
parents, and community members came forward and formed school plan- 
ning committees. 

• Most oriented at least a portion of their school missions and curricula to- 
ward a specific academic, artistic, social justice, or professional theme. 


^(Day, Bitter, and Gomez (2011). 

‘New York City Department ofEducation (2013a). 

3 Based on New York City Department ofEducation (2013b) four-year high school graduation rates for the 
cohort of students that entered ninth grade the year before their school began to be phased out. 

4 Fifty of these schools are in the Bronx and 31 are in Brooklyn. 

5 Other new small secondary schools created by New York City during this period include 38 general high 
schools for grades 9-12, which screen students based on their academic backgrounds; 21 transfer schools that 
are designed to help students who are overage for their grade and have fewer credits than they should; 33 
middle/high schools for grades 6-12 or 7-12; and 1 specialized high school for high-performing students. 
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New York City Small Schools of Choice 
Figure 1 

Location of SSCs That Were Opened and Large, Failing High Schools That Were Closed 

Between the Fall of 2002 and the Fall of 2008 
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• Most were founded in partnership with a local nonprofit organization or 
private employer that offered students relevant learning opportunities inside 
and outside the classroom, and that provided the SSC with additional staff 
support and resources. 

• Almost all were provided additional philanthropic funds during their first 
four or five years of start-up and were offered special policy protections by 
the NYCDOE during their first two start-up years, most notably exemp- 
tions from having to enroll special education students and English language 
learners. 

• Finally, almost all were started with the support of an intermediary school 
partner, such as New Visions for Public Schools, the Institute for Student 
Achievement, or the Urban Assembly. 6 

In June 2010, MDRC released a report on the effectiveness of 105 of the 123 new SSCs 
(all of those for which a rigorous analysis was possible). 7 That report demonstrated that the 
SSCs studied substantially improved academic progress and graduation rates for their students. 
Findings in the report reflected the experience of more than 2 1 ,000 first-time ninth-graders who 
entered high school between the fall of 2005 and the fall of 2008. These findings were based on 
a series of randomized lotteries used to determine who is assigned to SSCs when they have 
more applicants than seats. SSC lotteries are a byproduct of the districtwide High School 
Application Processing System (HSAPS). 8 

In a large sample, like that used for the MDRC study, lottery winners and lottery losers 
are in all ways the same, on average, before they enter high school. Consequently, it is valid to 
attribute any differences in their future academic outcomes to their access to an SSC. Because 
students who lose an SSC lottery attend over 200 widely varying high schools, the effectiveness 


6 Together these three intermediary partners 8 New Visions for Public Schools, the Institute for Student 
Achievement, and the Urban Assembly 3 were affiliated with roughly 70 percent of SSCs (approximately 50 
percent, 10 percent, and 10 percent respectively). For a detailed description of New Vision® role in support- 
ing around half of all SSCs through the New Century High Schools Initiative, see New Visions for Public 
Schools (2007). 

7 Bloom, Levy Thompson, andUnterman (2010). 

8 SSC lotteries are not public events, but rather the result of HSAPS student assignment rules. Appendix A 
describes how HSAPS creates SSC lotteries; how an SSC lottery creates a randomized treatment group of SSC 
lottery winners and a randomized control group of SSC lottery losers; and how this information, together with 
information about the schools in which students enrolled, was used to estimate the effects of enrolling in an 
SSC relative to enrolling in some other New York City public high school. Corresponding findings for the 
effects of winning an SSC lottery (but not necessarily enrolling in an SSC) are presented in Appendix B. These 
latter findings, which are less interpretable, are smaller in magnitude but consistent in nature with those for the 
effects of enrolling in an SSC. 
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of SSCs was judged against that of a diverse group of other high schools. 9 Because the large, 
failing schools that were closed no longer existed, the MDRC analysis could only compare the 
perfonnance of SSCs to that of the other schools that remained or were created after the large, 
failing high schools were closed. 10 

Data available in 2010 made it possible to follow four annual student cohorts through at 
least one year of high school and to follow one cohort through four years of high school. 
Findings in that report indicate that enrolling in an SSC instead of another type of New York 
City public high school markedly increased students’ progress toward graduation in their early 
high school years (based on data for all four cohorts) and also markedly increased their gradua- 
tion rates four years later (based on data for the earliest cohort). These findings also indicated 
that SSC effects on high school graduation were produced mainly by an increase in the propor- 
tion of students who received a New York State Regents diploma, with little or no effect on the 
proportion who received a local diploma (which has less stringent requirements and was phased 
out subsequently) or an advanced Regents diploma (which has more stringent requirements and 
is received by very few students who apply to SSCs). 

In January 2012, MDRC updated this analysis by adding graduation findings for a sec- 
ond student cohort. These results indicated that SSCs increased graduation rates for the second 
cohort and for many student subgroups within the two cohorts combined. 1 1 The findings also 
confirmed that SSC effects on high school graduation are produced mainly by an increase in the 
proportion of students who receive a Regents diploma. In addition, these findings indicated that 
SSC graduation effects were sustained after five years from the time sample members entered 
high school (based on data for the earliest cohort). 

Findings from both reports were widely discussed in national and local forums. How- 
ever, a number of important questions remained to be answered as additional follow-up data 
became available. This report addresses the following three questions: 

• Do SSCs’ substantial positive effects on students’ four- and five-year gradua- 
tion rates hold up over time as additional cohorts of students enter them and 
as graduation rates are increasing districtwide? 


9 Table 3.1 in Bloom, Levy Thompson, and Unterman (2010) compares many of the school, student, and 
teacher characteristics of SSCs with those of schools attended by control group members in the MDRC study. 

10 Because this study examines only one part of New York City’s high school reform initiative (small 
schools of choice), it does not attempt to determine the overall effect of all aspects of the initiative. Neither 
does it attempt to determine the effect of closing the 3 1 large, failing high schools. 

1 'Bloom and Unterman (2012). 
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• Do SSCs produce comparable positive effects for all student subgroups, in- 
cluding students who are eligible for special education services and students 
who are eligible for English language learner services? 

• What do principals and teachers from SSCs with the strongest evidence of ef- 
fectiveness think makes their schools effective, and what major problems did 
they encounter while creating and operating them? 

Findings in this report are based on a modified version of the sample used for MDRC’s 
earlier SSC reports. Appendix A describes how this modification simplifies and strengthens the 
methodology used. This modification does not appreciably affect the findings obtained. In brief, 
this report’s findings are: 

• SSCs in New York City continue to markedly increase high school gradua- 
tion rates for large numbers of disadvantaged students of color, even as grad- 
uation rates are rising at the schools with which SSCs are compared. 

• The best evidence that exists indicates that SSCs may increase graduation 
rates for two new subgroups for which findings were not previously availa- 
ble: special education students and English language learners. However, giv- 
en the still-limited sample sizes for these subgroups, the evidence will not be 
definitive until more student cohorts can be added to the analysis. 

• Principals and teachers at the 25 SSCs with the strongest evidence of effec- 
tiveness believe that academic rigor and personal relationships with students 
contribute to the effectiveness of their schools. They also believe that these 
attributes derive from their schools’ small organizational structures and from 
the commitment, knowledge, dedication, and adaptability of their teachers. 


Do SSC Effects Hold Up for a New Student Cohort? 

Findings in Tables 1-4 demonstrate that the positive SCC effects previously reported for the 
study’s first two student cohorts are sustained when graduation results are added for a third 
cohort, students who entered ninth grade in the fall of 2007. 12 

Four-Year SSC Graduation Effects by Student Cohort 

Findings in Table 1 indicate that, on average, for students in the first three cohorts, en- 
rolling in an SSC increased four-year graduation rates by 9.5 percentage points (to 70.4 percent 

12 Appendix A describes how these findings were obtained and are reported. 
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New York City Small Schools of Choice 
Table 1 


Estimated Effects of SSCs on Four-Year High School Graduation Rates 

by Student Cohort: 

Cohorts 1-3 


Cohort 

Target SSC 
Enrollees (%) 

Control 

Group Estimated 
Counterparts (%) Effect (%) 

P- Value for 
Estimated 
Effect 

Sample 

Size 

Cohort 1 (2004-2005) 

66.6 

58.3 

8.3 * 

0.014 

4,473 

Cohort 2 (2005-2006) 

70.4 

59.2 

11.2 ** 

0.000 

3,995 

Cohort 3 (2006-2007) 

74.6 

65.1 

9.5 ** 

0.001 

3,662 

Cohorts 1-3 

70.4 

60.9 

9.5 ** 

0.000 

12,130 


SOURCES: MDRC’s calculations use High School Application Processing System data for 
eighth-graders in 2004-2005 (cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus 
data from the New York City Department of Education’s fdes on student attendance, course 
credits. Regents examination scores, administrative transactions, and enrollment for the 2005- 
2006 to 2010-2011 school years. 

NOTES: Findings in this table are based on four-year follow-up data for 12,130 participants in 
192 lotteries for 84 SSCs. Estimates of the effect of enrolling in an SSC were obtained by 
comparing mean outcomes for winners and losers of students’ first SSC lottery while accounting 
for which lottery participants enrolled in an SSC and which did not, using the lottery outcome 
interacted with a binary lottery indicator as an instrumental variable for SSC enrollment and 
adjusting estimated standard errors for student clustering by the first school they attended (see 
Appendix A). 

A two-tailed t-test was used to assess the statistical significance of each estimated SSC effect 
with significance levels indicated as ** = 1 percent and * = 5 percent. Variation in estimated SSC 
effects across the three cohorts was not statistically significant (p-value for chi-square test = 
0.798). 


for target SSC enrollees — students for whom SSC effects were estimated — from 60.9 percent 
for their control group counterparts). 13,14 Across cohorts, this estimated effect ranges from 8.3 to 

l3 This finding is based on four-year follow-up data for 81 percent of the students in the study’s first three 
cohorts. The remaining 19 percent lack follow-up data because they moved out of the district or dropped out of 
school unofficially, or for other reasons. Appendix A demonstrates that SSC lottery winners and losers are 
equally likely to lack four-year follow-up data. In addition, the appendix demonstrates that lottery winners and 
losers among students who have this follow-up data are virtually identical, on average, with respect to a broad 
range of baseline characteristics. Thus the missing follow-up data are very unlikely to bias the present findings. 
This conclusion is further reinforced by the fact that even if one assumed that no students with missing data 
graduated from high school in four years, the estimated effect of SSCs would still be 8.5 percentage points 

(continued) 
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11.2 percentage points. 15 Thus SSC effects on four-year high school graduation rates were 
successfully replicated across three large samples. This result is especially striking given that 
graduation rates have been rising districtwide, and thus at the schools against which SSCs are 
compared. Kemple (2013) provides evidence of this rising tide, as does the fact that graduation 
rates for control group counterparts in Table 1 increase from 58.3 for the first cohort to 65.1 
percent for the third cohort. 

Four-Year SSC Effects on Graduation by Diploma Type and College 

Readiness 

During the present study period, students in New York State could receive one of three 
types of high school diplomas: a local diploma (which had the least stringent requirements), a 
New York State Regents diploma (which is generally considered to be the standard graduation 
credential), or a New York State Advanced Regents diploma (which has the most stringent 
requirements). Table 2 indicates that most of the increase in graduation rates caused by SSCs is 
due to a 6.0 percentage-point increase in the proportion of students who received New York 
State Regents diplomas. To obtain this type of diploma, students must score at least 65 points out 
of 100 on each of five required Regents examinations (English Language Arts, Mathematics, 
Science, Global History, and American History) and pass all courses required by the state. A 
much smaller portion of the increase in graduation rates caused by SSCs is due to a 2.3 percent- 
age-point increase in the proportion of students who received local diplomas, which were phased 
out for future student cohorts. An even smaller portion of the increase in graduation rates caused 
by SSCs is due to a 1.2 percentage-point increase in the proportion of students who received 
Advanced Regents diplomas, which are received by very few students who apply to SSCs. 

In addition, Table 2 indicates that enrolling in an SSC increased students’ college readi- 
ness in English, as measured by the percentage that scored at least 75 points on the English 
Regents examination. This threshold is used by the City University of New York (CUNY) to 
exempt incoming students from taking remedial English. Enrolling in an SSC increased this rate 
by 6.8 percentage points (to 40.2 percent for target SSC enrollees from 33.4 percent for their 
control group counterparts). In contrast, enrolling in an SSC had no effect on college readiness 
in mathematics as measured by the percentage of students that scored at least 75 points on their 


(although the percentage of students graduating from high school would be lower for both target SSC enrollees 
and their control group counterparts). 

14 Appendix A describes how target SSC enrollees are defined and how SSC effects are estimated for them. 

15 Although estimates of SSC effects are statistically significant for each student cohort, their variation 
across cohorts is not statistically significant at the conventional 0.05 level. 
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New York City Small Schools of Choice 
Table 2 

Estimated Effects of SSCs on 

Four- Year High School Graduation and College Readiness: 

Cohorts 1-3 


Outcome (%) 

Target 

SSC 

Enrollees 

Control 

Group 

Counterparts 

Estimated 

Effect 

P-Value for 
Estimated 
Effect 

Graduation 

Graduated from high school 

70.4 

60.9 

9.5 ** 

0.000 

Local diploma granted 

17.0 

14.7 

2.3 

0.145 

Regents diploma granted 

45.9 

39.9 

6.0 ** 

0.007 

Advanced Regents diploma granted 

7.5 

6.3 

1.2 

0.469 

College readiness 

English Regents exam score of 75 or above 

40.2 

33.4 

6.8 ** 

0.002 

Math A Regents exam score of 75 or above 

24.6 

24.7 

0.0 

0.989 


SOURCES: MDRC’s calculations use High School Application Processing System data for eighth- 
graders in 2004-2005 (cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus data from the 
New York City Department of Education’s files on student attendance, course credits, Regents 
examination scores, administrative transactions, and enrollment for the 2005-2006 to 2010-201 1 
school years. 

NOTES: Findings in this table are based on four -year follow-up data for 12,130 participants in 192 
lotteries for 84 SSCs. Estimates of the effect of enrolling in an SSC were obtained by comparing 
mean outcomes for winners and losers of students’ first SSC lottery while accounting for which 
lottery participants enrolled in an SSC and which did not, using the lottery outcome interacted with a 
binary lottery indicator as an instrumental variable for SSC enrollment and adjusting estimated 
standard errors for student clustering by the first school they attended (see Appendix A). Some 
findings may not sum exactly due to rounding error. 

A two-tailed t-test was used to assess the statistical significance of each SSC estimated effect with 
significance levels indicated as ** = 1 percent and * = 5 percent. 


Regents mathematics examination. 16 In addition, target SSC enrollees were much more likely to 
be college ready in English than in mathematics (40.2 percent versus 24.6 percent). 


16 ln 2012, CUNY raised its threshold for exempting students from taking remedial mathematics to 80 
points. The estimated SSC effect on the percentage of students who surmounted this threshold is 0.0 percent- 
age points. 






Five-Year SSC Graduation Effects by Student Cohort and Diploma Type 

Some students may take longer than four years to meet the local or state requirements to 
graduate high school. For example, about 5 percent of the students that began ninth grade in the 
falls of 2005, 2006, and 2007 took five years to graduate. 17 With this in mind, Table 3 presents 
estimates of SSC effects on high school graduation and college readiness after five years of 
follow-up. Findings in the top panel indicate that, on average, SSCs increased five-year gradua- 
tion rates for students in the first two cohorts (those for which five years of follow-up data are 
currently available) by 8.9 percentage points (to 73.6 percent for target SSC enrollees from 64.7 
percent for their control group counterparts). 18 

Estimates of this SSC effect on each cohort (7.1 and 10.3 percentage points) are about 1 
percentage point smaller than the corresponding estimates after four years. This reflects the fact 
that during the fifth follow-up year, graduation rates for target SSC enrollees rose by between 5 
and 6 percentage points while those for their control group counterparts rose by between 6 and 7 
percentage points. 

Findings in the bottom panel of Table 3 indicate that the effect of SSCs on five-year 
graduation rates is due primarily to an increase in the proportion of students who received 
Regents diplomas, as was the case for their four-year effect. After five years, therefore, the 
impact of SSCs remains roughly the same, as graduation rates rise similarly for both SSC 
enrollees and their control group counterparts. 

Four-Year SSC Graduation Effects by Student Subgroup 

Findings in Table 4 for subgroups in the study’s first three student cohorts indicate that 
SSCs increased four-year graduation rates for many different types of students, as was reported 
previously for the first two cohorts. 19 ’ 20 


17 New York City Department of Education (2013b). 

18 Five-year follow-up data are not yet available for cohort 3. 

19 To estimate SSC effects for a student subgroup, the sample for each SSC lottery was stratified by the 
student characteristic that defines the subgroup. Because not all lotteries include students from each stratum, 
different subgroup samples represent different combinations of lotteries. 

20 There is currently a movement among researchers and research funders to adjust statistical significance 
tests to account for the multiplicity of hypotheses tested by a given study. A number of statistical methods have 
been created for doing so — for example, the Bonferroni adjustment (Shaffer, 1995), the Benjamini-Hochberg 
adjustment (Benjamini and Hochberg, 1995), and the Westfall- Young adjustment (Westfall and Young, 1993). 
When a study estimates program effects for many different outcomes, time periods, or subgroups, many more 
chances exist to randomly obtain an apparently significant result. The proposed adjustments are designed to 
protect against this possibility of mistakenly judging a finding to represent a true program effect when it readily 
could have occurred by chance. This is not, however, a problem for the present subgroup analysis because 20 
of the 28 subgroup estimates are statistically significant and most of them (especially those for large samples) 

(continued) 


9 



New York City Small Schools of Choice 
Table 3 

Estimated Effects of SSCs on Five-Year Graduation Rates 
by Student Cohort and Diploma Type: 

Cohorts 1-2 


By Cohort or Diploma Type (%) 

Target SSC 
Enrollees 

Control 

Group 

Counterparts 

Estimated 

Effect 


P-Value for 
Estimated 
Effect 

Sample 

Size 

Bv cohort 







Cohort 1 (2004-2005) 

71.4 

64.3 

7.1 

* 

0.027 

4,500 

Cohort 2 (2005-2006) 

76.1 

65.9 

10.3 

** 

0.000 

4,021 

Cohorts 1-2 

73.6 

64.7 

8.9 

** 

0.000 

8,521 

Bv diploma type for cohorts 1-2 







Local diploma 

23.1 

21.0 

2.2 


0.320 

8,521 

Regents diploma 

44.1 

38.0 

6.2 

* 

0.011 

8,521 

Advanced Regents diploma 

6.3 

5.6 

0.7 


0.654 

8,521 


SOURCES: MDRC’s calculations use High School Application Processing System data for eighth-graders in 
2004-2005 (cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus data from the New York City 
Department of Education’s files on student attendance, course credits, Regents examination scores, administra- 
tive transactions, and enrollment for the 2005-2006 to 2010-2011 school years. 

NOTES: Findings in this table are based on five-year follow-up data for 8,521 participants in 143 lotteries for 82 
SSCs. Estimates of the effect of enrolling in an SSC were obtained by comparing mean outcomes for winners 
and losers of students’ first SSC lottery while accounting for which lottery participants enrolled in an SSC and 
which did not, using the lottery outcome interacted with a binary lottery indicator as an instrumental variable for 
SSC enrollment and adjusting estimated standard errors for student clustering by the first school they attended 
(see Appendix A). Some findings may not sum exactly due to rounding error. 

A two-tailed t-test was used to assess the statistical significance of each estimated SSC effect with signifi- 
cance levels indicated as ** = 1 percent and * = 5 percent. The difference between estimated SSC effects for 
cohorts 1 and 2 is not statistically significant (p-value for t-test = 0.463). 


are highly statistically significant. Indeed, if these 28 estimates were independent of each other (they are not, 
because most subgroups are overlapping) the probability of getting at least 20 estimates that are statistically 
significant at the 0.05 level (which is the significance threshold used for the present project) when no sub- 
groups have true SSC effects is infinitesimally small (based on the cumulative binomial distribution). 
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New York City Small Schools of Choice 
Table 4 

Estimated Effects of SSCs on Four-Year Graduation Rates 
for Student Subgroups: Cohorts 1-3 


Control P-Value for 

Target SSC Group Estimated Estimated 

Student Characteristic (%) Enrollees Counterparts Effect Effect 

Sample 

Size 

8th-grade reading proficiency 





t 

Did not meet standards (level 1) 

40.3 

30.2 

10.1 

0.152 

817 

Partially met standards (level 2) 

68.1 

58.3 

g g ** 

0.000 

6,452 

Fully met standards (level 3) 

85.9 

74.7 

11.2 ** 

0.000 

4,348 

Met standards with distinction (level 4) 

88.5 

90.0 

-1.5 

0.698 

513 

8th-grade math proficiency 





t 

Did not meet standards (level 1) 

47.2 

36.9 

10.2 * 

0.043 

3,578 

Partially met standards (level 2) 

68.1 

56.3 

11.8 ** 

0.000 

5,707 

Fully met standards (level 3) 

88.4 

76.8 

11.5 ** 

0.000 

2,548 

Met standards with distinction (level 4) 

98.8 

O 

O 

P 

-1.2 

0.790 

297 

Low-income status 






Eligible for free/reduced-price lunch 

68.6 

57.4 

11.2 ** 

0.000 

7,418 

Not eligible for free/reduced-price lunch 

73.2 

66.2 

7.0 * 

0.011 

4,712 

Race/ethnicitv. bv gender 






Black male 

65.5 

52.0 

13.5 ** 

0.002 

2,300 

Black female 

72.8 

64.7 

8.0 * 

0.042 

2,917 

Flispanic male 

64.5 

57.2 

7.3 

0.068 

2,745 

Flispanic female 

73.2 

62.9 

10.3 ** 

0.002 

2,930 

Other male 

83.2 

77.6 

5.6 

0.197 

552 

Other female 

87.6 

78.1 

9.5 

0.154 

510 


(continued) 
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Table 4 (continued) 


Student Characteristic (%) 

Target SSC 
Enrollees 

Control 

Group 

Counterparts 

P-Value for 
Estimated Estimated 
Effect Effect 

Sample 

Size 

Known or unknown to SSC 






Known 

72.9 

64.9 

8.0 ** 

0.002 

6,823 

Unknown 

63.6 

51.6 

12.1 ** 

0.000 

5,307 

Choice level (of 12) at which enrollee participated 

in lottery 




1 st choice 

71.8 

63.3 

8.5 ** 

0.001 

5,688 

2 nd choice 

66.9 

60.0 

6.8 * 

0.045 

2,689 

All other choices 

70.2 

55.1 

15.2 ** 

0.000 

3,753 

Special education status 






Eligible for services 

62.8 

48.9 

13.8 

0.074 

725 

Not eligible for services 

70.7 

61.5 

g 3 h=h= 

0.000 

11,405 

English language learner 






Eligible for services 

63.9 

59.0 

4.9 

0.418 

843 

Not eligible for services 

70.8 

61.3 

9.5 ** 

0.000 

11,287 


SOURCES: MDRC’s calculations use High School Application Processing System data for 
eighth-graders in 2004-2005 (cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus data 
from the New York City Department of Education’s fdes on student attendance, course credits, 
Regents examination scores, administrative transactions, and enrollment for the 2005-2006 to 
2010-2011 school years. 

NOTES: Findings in this table are based on subgroups from a sample of 12,130 students who have 
four-year follow-up data and participated in 192 lotteries for 84 SSCs. Because not all subgroups 
are represented by all lotteries or SSCs, findings for different subgroups can represent a different 
mix of lotteries and SSCs. Estimates of the effect of enrolling in an SSC were obtained by 
comparing mean outcomes for winners and losers of students’ first SSC lottery while accounting 
for which lottery participants enrolled in an SSC and which did not, using the lottery outcome 
interacted with a binary lottery indicator as an instrumental variable for SSC enrollment and 
adjusting estimated standard errors for student clustering by the first school they attended (see 
Appendix A). Some findings may not sum exactly due to rounding error. 

A two-tailed t-test was used to assess the statistical significance of each estimated SSC effect 
with significance levels indicated as ** = 1 percent and * = 5 percent. A chi-square test was used 
to assess the statistical significance of variation in estimated SSC effects across subgroups within a 
given dimension, with significance levels indicated as ff =1 percent and f= 5 percent. 

a This value was truncated to exactly 100. 


12 






Subgroups Defined by Students’ Prior Academic Proficiency 

The first findings in the table are for subgroups defined by students’ prior academic 
proficiency, as measured by their eighth-grade performance on New York State tests of reading 
and math. Proficiency levels 1 and 2 are considered by the state to be below grade level, 
whereas proficiency levels 3 and 4 are considered to be at or above grade level. In eighth grade, 
roughly two-thirds of present sample members scored below grade level and thus were not fully 
prepared to do ninth-grade academic work when they entered high school. 

The findings indicate that SSCs markedly increased four-year graduation rates for stu- 
dents in the first three proficiency levels of both subjects, with estimated effects ranging from 
9.8 to 1 1.8 percentage points. This is especially striking given the enormous variation that exists 
in future graduation rates across eighth-grade proficiency levels (from 30.2 to 74.7 percent for 
reading and from 36.9 to 76.8 percent for math). However, SSCs have no effect on graduation 
rates for the few sample members who are in the top prior proficiency levels because almost all 
of the control group counterparts also graduate (100 percent in the top math subgroup and 90.0 
percent in the top reading subgroup). 

Subgroups Defined by Students’ Low-Income Status 

Findings for subgroups of students who were and were not eligible for free or reduced- 
price lunches tell a similar story. They indicate that SSCs increased graduation rates by 1 1 .2 
percentage points for students who are eligible for this subsidy and by 7.0 percentage points for 
students who are not eligible — even though their graduation prospects differ substantially 
(from 57.4 percentage points for control group counterparts who are eligible for free or reduced- 
price lunches to 66.2 percent for those who are not eligible). 

Subgroups Defined by Students’ Race/Ethnicity 

Findings by students’ race/ethnicity and gender provide further evidence that SSCs in- 
crease graduation rates for many different types of students. This is especially true for male and 
female Black and Hispanic students, who comprise the largest samples in this set of subgroups. 
Estimates for these four subgroups are positive and three of the four are statistically signifi- 
cant. 21 Estimates for “other” male and female students are also positive, although their small 
sample sizes limit the statistical significance of these results. 


“'The finding for Hispanic male students, which is not statistically significant at the 0.05 level (the thresh- 
old used for the present study), is statistically significant at the 0.10 level (a threshold used by many researchers 
and academic journals). 
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Subgroups Defined by Whether or Not Students Were Known to Their 

SSCs When They Applied 

The next findings in the table are for subgroups of students who were or were not 
known ahead of time to the SSCs in whose lotteries they participated. 22 A student can become 
known to an SSC by contacting it in person or by telephone, by visiting it, by meeting with its 
representatives at a high school fair, or in other ways. To promote informed school choice, the 
NYCDOE encourages eighth-graders and their parents to do all of these things. 

The findings indicate that sample members who were known to their SSCs experienced 
an 8.0 percentage-point increase in four-year graduation rates, while sample members who were 
not known to their SSCs experienced a 12.1 percentage-point increase. It is interesting to note 
that students who make the effort are more likely to graduate than students who do not make the 
effort regardless of whether or not they attend an SSC. 

Subgroups Defined by Students’ Rank-Ordered Preference for Their SSC 

The next subgroups in the table are defined by the rank-ordered preference that students 
placed on the SSC in whose lottery they participated. The first subgroup represents students 
whose first SSC lottery was for their first-choice school, the next subgroup represents students 
whose first SSC lottery was for their second-choice school, and the final subgroup represents 
students whose first SSC lottery was for a school that was one of their third through twelfth 
choices. Positive SSC effects ranging from 6.8 to 15.2 percentage points are reported for all 
three of these subgroups. Thus, contrary to what might be expected, students who attend their 
first-choice SSC do not experience SSC effects that are larger than those experienced by 
students who attend a less preferred SSC. 


Do SSCs Produce Positive Graduation Effects for New Student 
Subgroups? 

As noted previously, adding a third student cohort to the sample made it possible to report 
preliminary graduation findings for two new sets of subgroups: (1) students who were and were 
not eligible for special education services and (2) students who were and were not eligible for 
English language learner (ELL) services. 23 These subgroups are important for several reasons. 


22 Appendix A describes how each SSC lottery includes either students who were known to the SSC or 
students who were not known to it but not both. Lottery winners and losers are thus perfectly matched in this 
regard. 

23 SSC lotteries include special education students who can be taught in a regular classroom setting. Special 
education students classified by the NYCDOE as requiring collaborative team teaching services or self-contained 

(continued) 
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First, early SSCs were exempted by the NYCDOE from serving special education stu- 
dents and English language learners during their first two years of operation (some did, some 
did not). This exemption reflected a concern that serving students with special needs requires 
expertise and resources that might not be available when schools are starting up. However, the 
exemption, which was a source of contention, was later rescinded. Second, it is widely per- 
ceived that few education interventions have produced sizable gains at a large scale for students 
with special needs, even though these students are present in large numbers in many school 
districts and even though these districts are assessed under No Child Left Behind in part by the 
performance of their special-needs students. Third, many educators and parents feel that special- 
needs students are often left out of broad-based education policy interventions such as school 
choice systems, charter schools, and targeted college-readiness programs. 

Subgroups Defined by Eligibility for Special Education Services 

Findings in Table 4 suggest that SSCs increased graduation rates for special education 
students by 13.8 percentage points (to 62.8 percent for target SSC enrollees from 48.9 percent 
for control group counterparts). However, because of the subgroup’s still-limited sample size, 
this estimate is not statistically significant and thus is subject to more uncertainty than are 
estimates for most other subgroups (which have much larger samples). 24 Existing evidence 
therefore suggests that enrolling in an SSC increases graduation rates for special education 
students, but this finding is not yet definitive. Stronger evidence for this subgroup will require 
four-year follow-up data for additional student cohorts. 

It is also worth noting that 15.4 percent of all first-time HSAPS enrollees at the study’s 
SSCs were special education students, versus 13.9 percent districtwide. 25 Thus, even though 
some SSCs were exempt from admitting special education students during their earliest start-up 
years, SSCs in the study enrolled these students at rates that were roughly proportional to their 
size on average. 


classes are not part of SSC lotteries, although they attend SSCs at roughly the same rate as the average NYCDOE 
high school. In addition, the sample is limited to students who participate in the HSAPS process. 

~ 4 Although this estimate is not significant at the 0.05 level (the threshold used for the present study), it is 
significant at the 0.10 level (p-value = 0.074), a threshold used by many researchers and academic journals. 

25 Because special education students who cannot be taught in a regular classroom setting are not part of 
SSC lotteries, the findings reported for special education students do not apply to these students even though 
they also attend SSCs. Therefore the percentage of target SSC enrollees who are special education students (6.0 
percent) is smaller than the corresponding percentage of all first-time SSC enrollees (15.4 percent). 
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Subgroups Defined by Eligibility for English Language Learner Services 

Findings in Table 4 indicate that SSCs increased graduation rates for English language 
learners by 4.9 percentage points (to 63.9 percent for target SSC enrollees from 59.0 percent for 
control group counterparts). This finding is not statistically significant and is smaller than that 
for special education students. Thus, although the estimate suggests that SSCs improve gradua- 
tion prospects for English language learners, this evidence is weaker than that for most other 
subgroups, including special education students. 

Lastly, it is useful to compare the percentage of all first-time enrollees at the study’s 
SSCs who were English language learners (8.3) with the corresponding percentage for all 
NYCDOE high schools (11.2). This indicates that SSCs in the study sample are enrolling 
English language learners somewhat less often than they would if they were doing so in 
proportion to their size. 


What Do Principals and Teachers from Effective SSCs Think 
Make Their Schools Effective, and What Major Problems Have 
They Encountered? 

Although each SSC has its own mission and educational approach, they all are operated by the 
NYCDOE and staffed by union teachers and administrators (unlike most charter schools), and 
they all were created between the fall of 2002 and the fall of 2008 through a competitive and 
demanding proposal process (unlike many small schools created earlier in New York City and 
elsewhere). In addition, until 2008, planning teams that were selected to create a small school 
could receive $100,000 per year for up to five years (one planning year and four years of 
operation) from a consortium of funders, as long as they worked with an intermediary organiza- 
tion or community partner to create their school. 26 Often these partners were on the original 
proposal team and many had expertise in starting schools, implementing focused curricula, and 
linking students’ in-school work with out-of-school experiences. 

The SSC proposal process required teachers, administrators, community members, par- 
ents, and intermediary organizations to create new schools that were based on the core princi- 
ples of “[academic] rigor, relevance, and relationships,” and each proposal was evaluated 
according to its likelihood of enacting these principles. For example, in 2008, SSC proposals 
were assessed with respect to 10 elements, almost all of which are directly related to the 
model’s core principles (see Appendix C). The following are three examples of these criteria: 


26 The Bill & Melinda Gates Foundation was the lead funder in this consortium. In addition, a small 
amount of Bill & Melinda Gates Foundation funding was given to charter-based small schools (which are not 
part of the present analysis) in 2009. 
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(4) High expectations for all students and a standards-based, academically rigor- 
ous curriculum that connects what students leam with college and career goals 


(9) Connections between what students leam in school to their lives and com- 
munities through internships, mentoring experiences, and service learning op- 
portunities 

(10) A structure that fosters the development of authentic, sustained, caring, and 
respectful relationships between teachers and students and among staff mem- 
bers 

Given the emphasis placed on these principles by SSC funders and the stated intentions 
of SSC proposers to enact them, MDRC sought to determine the extent to which they are 
reflected in practice. For this purpose, MDRC contracted with the Research Alliance for New 
York City Schools at New York University to conduct interviews with principals and focus 
groups with teachers from a sample of SSCs that had the strongest evidence of effectiveness, to 
leam their views on the importance of these principles and other educational practices for their 
schools’ effectiveness. The Research Alliance also conducted a broad-based survey of teachers 
in as many as possible of the 105 SSCs studied by MDRC to leam from them about variation in 
the role of these principles and other educational practices. Findings from the survey will be 
used in a future report. 

For the interviews and focus groups, MDRC chose a target sample of 30 SSCs. 27 The 
Research Alliance designed the fieldwork protocols, conducted the fieldwork at 25 of these 
SSCs during the 2011-2012 school year, prepared a data set from the results, and conducted 
initial analyses of these data. 28 At 23 SSCs the current principal was interviewed and at two 
SSCs the current assistant principal was interviewed. Two-thirds of these interviewees had 
joined their SSC in its first year of operation. Focus groups typically comprised three to five 


27 MDRC chose the 30 SSCs that had the largest and/or most statistically significant positive estimates of 
effects on ninth-graders’ progress toward graduation during the two most recent school years for which these 
data were available at the time (2008-2009 and 2009-2010). Footnote 2 in Appendix D describes how these 
schools were selected in greater detail. The 30 targeted SSCs do not necessarily represent all effective SSCs. 
Some SSCs did not have student lotteries for the years examined (thus it was not possible to assess their 
effectiveness), and others had lottery samples that were too small to provide strong evidence about their 
effectiveness. In addition, because estimates of effectiveness used to target SSCs contain error (like any 
estimate), it is likely that not all targeted SSCs were, in fact, effective. However, there is strong evidence that 
the targeted group of SSCs is highly effective on average. 

28 After a vigorous recruitment effort by the Research Alliance, interviews and focus groups were conduct- 
ed at 25 of the 30 targeted SSCs. Principals at two SSCs declined to have their schools participate because of 
concerns about time limitations, and principals at three SSCs did not respond to the numerous attempts to 
recruit them. Appendix D provides more detailed information about the interviews with SSC principals and the 
focus groups with their teachers. 
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teachers, with most including a founding teacher and a majority of participants who had been at 
their current school for at least four years. Two main questions were addressed: 

• What features of these SSCs do the professionals who created and operate 
them think are most responsible for their effectiveness? 

• What do these professionals think are the most serious challenges to creating 
and maintaining the effectiveness of their SSC? 

Answers to these questions can help to: (1) assess the extent to which effective SSCs 
embody the model® core principles, (2) provide insights into what the people who know the 
most about these schools think makes them effective (which is an important source of hypothe- 
ses for future research), and (3) provide information about challenges that others who are 
considering creating SSCs should anticipate. 

This section presents a brief summary of the results from the interviews and focus 
groups. Future MDRC research on factors that influence variation in the effectiveness of SSCs 
will use (among other things) data collected by the Research Alliance from its survey of 
teachers at 86 of the study® original 105 SSCs plus publicly available information obtained by 
annual NYCDOE surveys of teachers, students, and parents at all high schools operated by the 
district. What is reported below should be considered a nfirst peeko inside the black box of SSC 
effectiveness. 

What Factors Do SSC Principals and Teachers Think Promote Their 

Schools’ Effectiveness? 

Early in the interviews and focus groups, participants were asked the following open- 
ended question: nOf all of the things that contribute to making a school work, what are the two 
or three factors you think are most responsible?OTable 5 summarizes responses to this question. 
The first column reports the percentage of SSCs whose principal mentioned each factor, and the 
second reports the percentage of SSCs whose principal mentioned each factor first. The third 
and fourth columns report corresponding findings for teacher focus groups. 

As can be seen, the most frequent responses (and first responses) from both principals 
and teachers are high-quality teachers, personalized learning environments ( relationships ), and 
high academic expectations (rigor). Responses about the third core SSC principle, relevance, 
were less frequent and more mixed. 

High-Quality Teachers 

Principals described teachers as educational leaders who drive their school® personal- 
ized environment and rigorous curriculum and often are called upon to assume leadership 
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New York City Small Schools of Choice 
Table 5 

Factors Perceived to Influence SSC Effectiveness 

Principal Interviews Teacher Focus Groups 


Factor Factor Factor Factor 

Factor (%) mentioned mentioned first mentioned mentioned first 


Teachers 

84 

52 

64 

8 

Personal relationships 

76 

28 

88 

52 

High academic expectations 

24 

4 

80 

12 

Leadership 

20 

- 

36 

12 

Collaboration 

16 

8 

- 

- 

External partners 

4 

- 

16 

- 

Data use/tracking students 

- 

- 

12 

8 


SOURCE: Interviews with principals and focus groups with teachers conducted for MDRC by 
the Research Alliance for New York City Schools at 25 SSCs with the strongest evidence of 
positive effects on the percentage of students who were on track toward graduation in ninth 
grade during the two most recent school years for which these data were available at the time 
(2008-2009 and 2009-2010). At 23 SSCs the current principal was interviewed and at two SSCs 
the current assistant principal was interviewed. Two-thirds of these interviewees had joined their 
SSC in its first year of operation. Focus groups typically comprised three to five teachers, with 
most including a founding teacher and a majority of participants who had been at their current 
SSC for at least four years. Findings in the table represent responses to the following open- 
ended question: “Of all the things that contribute to making a school work, what are the two or 
three factors you think are most responsible?” 


positions and responsibilities much sooner than typically is the case elsewhere. For example, 
one principal noted that teaching in his SSC requires a steadfast dedication to the school and 
continuous rigorous assessment of what is and is not working for students: 

With our teachers, there’s a desire to come together, but there’s also that ded- 
ication to the kids and that intellect that allows them to analyze what the 
problems are. 

Another principal echoed this sentiment: 

The instructional program is very, very important, so really creating a pro- 
gram where you are constantly developing a system of self-reflection and 
self-regulation is really important. 
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Similarly, an SSC teacher noted that she is given the space to innovate independently, 
as long as she can quickly feed results back to her principal: 

We are allowed as teachers in our classroom to try anything, as long as we 
can prove that it works.... I’ve been able to have double periods for the last 
four years because I’ve been pushing it with my principal and she lets us do 
it. I’ve been able to push essay writing as a real heavy focus in my class, and 
I never get a complaint from the principal as long as I can prove that it’s 
working. 

Many principals described their teachers as being flexible, dedicated, and driven, and 
noted that when hiring new teachers they give weight to these traits and to a candidate’s likely 
compatibility with their school’s environment. Although different principals had different 
approaches to hiring staff, the majority used a committee-based approach that drew heavily on 
their lead teachers. All principals viewed their staffing process as the point at which they began 
to create their school environment. For example, one principal commented: 

One of the most important things is getting the right people in the right posi- 
tions, and that takes time. Making sure that you hire the best teachers, the 
most highly qualified teachers who know their content, who have a love for 
kids, who are quite intelligent, who can implement feedback well. 

Personalized School Environments 

Principals described their schools’ personalized environments as settings in which staff 
know all students and have their trust. They noted that staff members draw on the small size of 
SSCs to create environments where they can effectively track student performance and prevent 
them from “slipping through the cracks.” One principal noted that: 

The seniors come to me on a regular basis, and I have a whole stack of what 
everyone needs to graduate. [They ask me,] “Miss, what do I need to gradu- 
ate?” ... [We say,] “Okay, you need to do this, this and that.” And when we 
see them in the hallway, [we say,] “Okay, did you do that? Did you do this?” 

It’s kind of a motivational thing for them. They know that somebody is tak- 
ing a vested interest in them. 

One feature of personalized school environments that was noted is their extension beyond 
purely academic issues to address needs of the whole student and their families. For example, one 
principal described having seen teachers and other school staff members help students that could 
not afford appropriate clothing to find it, or help parents to take advantage of existing community 
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resources. This was observed so often that the principal named it a “guidance intervention,” 
which, in his opinion, was a critical component of building a school community. 

Teachers in 88 percent of the focus groups mentioned their school’s personalized envi- 
ronment as a key to its effectiveness. In one teacher’s words: 

If it wasn’t for the interpersonal relationships that we develop with our stu- 
dents, I don’t think that teachers, or students, or families would have the 
same sort of buy-in that we’re able to cultivate. It’s very difficult to tell a 
teacher that you’ve been working with for four years, “I’m not going to be 
able to graduate because of this reason.” There’s always something that that 
teacher knows ... about the student or the family ... that brings out the best of 
her ability to make students want to earn their diploma. 

High Academic Expectations 

Teachers in 80 percent of the focus groups mentioned high academic expectations as a 
source of SSC effectiveness. Teachers in all but one of these focus groups also mentioned 
personalized environments. 29 Why are high academic expectations and personalized environ- 
ments so strongly connected in teacher’s minds? In conversation, many teachers attributed their 
ability to maintain high academic expectations to their ability to track students’ progress and 
know each student as an individual. For example, one teacher said: 

Our [school’s] personalized learning enviromnent and high academic expec- 
tations are the most important [features of the school that are responsible for 
its success]. You’ve got to show the kids what they’re capable of doing by 
expecting them to do it. Then, of course, the more personalized it is, the more 
you’re able to help them [succeed in their work]. 

Although teachers mentioned the importance of high academic expectations to the suc- 
cess of their students, they also indicated that it was difficult to consistently maintain these 
expectations with so many students entering high school performing below grade level (as 
noted in the section on challenges below). 

Reprise on the SSC Core Principles 

What do these findings say about the perceived importance of SSCs’ core principles: ri- 
gor, relevance, and relationships? First, they suggest that principals and teachers from the 25 
SSCs with the strongest evidence of effectiveness almost uniformly view relationships and rigor 


~ 9 This result, which is not reported in Table 5, was obtained directly from the fieldwork data. 
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as vital factors for their schools’ success. In addition, respondents consider these factors to be 
enabled by their school’s small size and its committed and well-aligned staff. 

However, views are more mixed about the third core SSC principle, relevance. One 
way schools might have put “relevance” into practice could have been through their academic, 
artistic, social-justice, or professional themes. Ideally, each school’s theme could have provided 
its students with an immediate purpose and reason for learning, helping them to connect what 
they were taught in school with their daily lives and future aspirations. But when Research 
Alliance researchers mentioned school themes in interviews and focus groups, reactions were 
mixed. Sometimes principals and teachers indicated that their school’s theme was important to 
its mission and curriculum, and other times they suggested it was not. For example, some 
respondents thought that their school’s theme limited its curriculum and made it difficult to 
focus on college readiness, while other respondents thought that the theme of their SSC differ- 
entiated it from other schools and thereby helped to attract teachers and students. 

Likewise, responses about SSCs’ external partners varied substantially. It is difficult to 
interpret these responses, however, because at times it was not clear whether respondents were 
referring to their Gates-funded community partner, an organization providing professional 
development and school support, or some other local collaborator. The Gates-funded partner 
often helped found the SSC and provided staff with additional resources during the first few 
years of operation. In addition, all NYCDOE high schools are required to provide professional 
development for teachers and other school supports, and beginning in the fall of 2007 principals 
were authorized to choose external organizations to provide these services from an NYCDOE- 
approved list. Schools also fonn partnerships with organizations that implement extracurricular 
activities or specific subject areas like science or history. 

Nevertheless, some respondents felt that their external partners promoted SSC success 
by giving students real-world opportunities to use what they were learning and providing a 
window into potential future careers. Other respondents felt that their external partners did not 
play a key role. In addition, when respondents mentioned the contribution of their school’s 
supporting organization(s), they often noted receiving support for specific tasks, like interpret- 
ing student data, facilitating relationships with the NYCDOE, receiving grant funding, or 
securing space — instead of directly providing learning opportunities for students. 

What Factors Do SSC Principals and Teachers Think Challenge Their 

Schools’ Effectiveness? 

Early in their interviews, when principals were discussing the creation of their SSC, 
they were asked “What were the major challenges the school faced during this period?” 
Toward the end of these interviews they were asked “What are the biggest challenges facing 
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this school today?” During focus groups, teachers were asked “What are the current challenges 
that this school is facing today? What challenges, if any, do you foresee this school confronting 
in the future?” Table 6 summarizes responses to these questions. 

Finances 

The most frequent response (and first response) by principals was a concern about fi- 
nancial resources, typically from two key sources — the NYCDOE and external partners. In the 
words of one principal: 

I think small schools, in general, are in a vulnerable position because we 
have fewer staff, fewer resources generally than bigger schools.... If I lose 
two college counselors, then my whole college advising program is gone. 

Principals were especially concerned that their external partners would cut back support 
in response to the sluggish postrecession economy. In the words of one principal: 

One other concern is I know that our partners are also facing these financial 
issues as well.... If our partners leave, then for the science department that 
would be a huge hit. 

Space 

In addition to financial concerns, many principals noted the day-to-day challenge of 
competing for space. For example, during start-up, principals often had to engage in power 
struggles with other principals who were operating schools in the same building. One princi- 
pal recalls: 


There was a lot of hostility amongst the small schools and [the large school 
on that campus being closed down] at the time, and it was really bad.... That 
was probably the biggest obstacle because you had to fight over space. 

Principals and teachers also noted their ongoing struggle to maintain and convey high 
academic expectations for students. Although many respondents viewed high academic 
expectations as part of what makes their SSC successful, they often felt it was difficult to 
maintain these expectations with so many students entering high school performing well 
below grade level. 

Staffing 

Another key challenge mentioned by principals and teachers was staffing. Since most 
SSCs began with only a ninth-grade class — and added a subsequent grade each year — they 
often did not have sufficient funding early on for a full complement of staff members, as most 
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New York City Small Schools of Choice 
Table 6 


Perceived Challenges to SSC Effectiveness 



Principal Interviews 

Teacher Focus Groups 


Challenge 

Challenge 

Challenge 

Challenge 

Challenge (%) 

mentioned 

mentioned first 

mentioned 

mentioned first 

Financial resources 

64 

28 

28 

24 

Space 

44 

20 

12 

- 

Academic expectations 

40 

16 

40 

16 

Staffing 

36 

- 

88 

24 

High-need student populations 

24 

4 

24 

- 

Attendance 

- 

- 

8 

8 

Socioeconomic challenges 

28 

12 

28 

4 

Discipline 

- 

- 

4 

4 

Other challenged) 

too 

20 

too 

20 


SOURCE: Interviews with principals and focus groups with teachers conducted for MDRC by the 
Research Alliance for New York City Schools at 25 SSCs with the strongest evidence of positive 
effects on the percentage of students who were on track toward graduation in ninth grade during the 
two most recent school years for which these data were available at the time (2008-2009 and 2009- 
2010). At 23 SSCs the current principal was interviewed and at two SSCs the current assistant 
principal was interviewed. Two-thirds of these interviewees had joined their SSC in its first year of 
operation. Focus groups typically comprised three to five teachers, with most including a founding 
teacher and a majority of participants who had been at their current SSC for at least four years. 
Findings in the table represent responses to the following open-ended questions: “What were the 
major challenges the school faced during this period [when it opened]?” “What are the biggest 
challenges facing this school today?” and “What challenges, if any, do you foresee this school 
confronting in the future?” 


funding is driven by student enrollment. Consequently, SSC staff members had to take on 
multiple roles. One teacher noted that: “The assistant principal of instruction not only supports 
teachers, but she’s also the school accountant.” 

Principals and teachers also expressed concern about staff turnover. In particular, they 
expressed concern that new staff members would not have the same level of commitment or 
ability to build trust and relationships as those who were there at the launch of the school. One 
teacher said: 


I think there’s a lot of changing going on [in staffing] that’s going to affect 
what happens. And I think because a lot of the initial people who started this 
school have now left or are leaving.... How do you keep the mission of a 
school alive and true if no one who started that said mission is here to fight 
for it? 
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What Has Been Learned and What Questions Remain? 

Findings from the present analysis indicate that: 


• SSCs in New York City continue to markedly increase high school gradua- 
tion rates for large numbers of disadvantaged students of color, even as grad- 
uation rates are rising at the schools against which SSCs are compared. This 
finding has now been replicated for three annual student cohorts, both on av- 
erage for the full study sample and separately for many different student sub- 
groups. 

• The best evidence that exists suggests that SSCs increase graduation rates for 
two new subgroups for which findings were not previously available: special 
education students and English language learners. However, given the still- 
limited sample sizes for these subgroups, the evidence will not be definitive 
until more student cohorts can be added to the analysis. Doing so will both 
increase the precision of future estimates of SSC effects on four-year gradua- 
tion rates and make it possible to estimate SSC effects on five-year gradua- 
tion rates. 

• Discussions with principals and teachers at the 25 SSCs with the strongest 
evidence of effectiveness provide evidence that two of the three SSC core 
principles — academic rigor and personal relationships — are deeply in- 
grained in the day-to-day operation of their schools. In addition, responses by 
these knowledgeable participant/observers indicate that they strongly believe 
that the two principles promote the effectiveness of their schools. Views are 
mixed, however, about the role and importance of the third SSC principle, 
relevance. 

• Principals and teachers at those 25 SSCs also provide evidence that commit- 
ted, knowledgeable, hardworking, and adaptable teachers are a key determi- 
nant of their schools’ effectiveness and that the small organizational struc- 
tures of the schools make it possible for teachers to enact the core principles 
of rigor and relationships. 

• Key challenges to the successful creation and ongoing operation of SSCs re- 
ported by their principals and teachers include: limited and shrinking finan- 
cial resources; initial problems with space when schools were being created; 
problems recruiting and keeping high-quality staff who can withstand the 
many taxing demands of their jobs; and problems maintaining high academic 
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expectations when many students enter high school operating well below 
grade level. 

These findings demonstrate that with sufficient commitment, resources, planning, and 
implementation it is possible to make much-needed progress toward improving a large urban 
high school system. However, more needs to be learned in order to keep improving the educa- 
tional outcomes of students in New York City (almost a third of whom still do not graduate) and 
to translate these findings into educational improvements elsewhere. 

First, it is essential to leam more about what makes SSCs effective so that these factors 
can be built into other initiatives. What organizational resources (such as past experience 
creating new schools), human resources (such as the existing labor pool for teachers and school 
administrators), financial resources (such as those provided by philanthropic donors), and 
political resources (such as the decision-making structure of a school district) are necessary to 
create effective SSCs? What must faculty and administrators in these schools do to sustain their 
effective operation? 

Although the well-informed opinions of SSC principals and teachers about these factors 
are a good source of hypotheses, they do not provide confinnation of their actual influence. To 
leam more, MDRC is conducting a statistical analysis of variation in effectiveness among SSCs 
and the factors that predict this variation — essentially, describing the factors that make the 
most successful SSCs effective. This work will use lottery-based estimates of SSC effectiveness 
plus data about the operation of SSCs obtained from teacher surveys conducted by the Research 
Alliance; data about the environment of all NYCDOE high schools obtained from annual 
surveys of teachers, students, and parents conducted by the district; and other information about 
the operation and environment of NYCDOE high schools from administrative records and 
publicly available sources. 

In addition, MDRC will determine whether the observed gains in high school gradua- 
tion produced by SSCs translate into observable gains in students’ success in postsecondary 
education and the world of work. MDRC will use data from the National Student Clearinghouse 
to estimate SSC effects on the extent to which current sample members enroll in and graduate 
from community colleges, professional schools, or four-year institutions. MDRC is also 
exploring a variety of approaches for obtaining data about future labor market outcomes for 
sample members. 

Given national attention on improving teacher quality and the consistently powerful 
message from SSC principals and teachers about the importance of high-quality teachers in their 
schools, MDRC is exploring ways to study the backgrounds, recruitment, and turnover of SSC 
teachers relative to that of teachers in other NYCDOE schools. 


26 



In order to help determine the likely feasibility and desirability of implementing SSC- 
like schools in other districts, it is important to leam more about the costs of creating and 
operating them. To do so, MDRC has contracted with an expert in school finance to study the 
operating cost structure of SSCs versus that of schools attended by SSC control group members. 
To the extent possible in this retrospective study, MDRC also will attempt to study the costs of 
creating SSCs. 

In closing, it is important to note that SSCs — like any other high school refonn — are 
not a panacea. Even though this study’s findings indicate that SSCs markedly increase gradua- 
tion rates for their students, it is still the case that many of them do not graduate from high 
school. Nonetheless, given the unusually large scale of the SSC intervention, the fact that it 
serves predominantly economically and educationally disadvantaged students of color, the 
robustness of its effects across many different student subgroups, and the fact that SSCs operate 
within an existing public school system and are staffed by union teachers and administrators, 
the small schools experience in New York City offers important lessons to help guide future 
refonns of public high school education. 
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Appendix A 

Sample, Data, and Analysis 




This appendix describes how the present sample was constructed and used to estimate the effects 
of enrolling in an SSC on students’ academic attainment and achievement. The first section 
describes how the second round of New York City’s High School Application Processing 
System (HSAPS) produces randomized lotteries for student assignment to SSCs. 1 The second 
section describes how these lotteries were used to construct the present sample. The third section 
describes how data for this sample were obtained. The fourth section describes how these data 
were used to estimate the effect of enrolling in an SSC on high school graduation. The fifth 
section describes how estimates of SSC effects are reported and how to interpret these findings. 
The sixth section examines the likely effect of sample attrition on these estimates. 


How HSAPS Produces SSC Lotteries 

HSAPS currently assigns more than 90 percent of New York City’s 90,000 annual entering 
ninth-graders to more than 400 public high schools. Students submit up to 12 high school 
choices in rank order, schools submit their priorities for students, and HSAPS assigns students 
to schools based on this information. Through this process, three-quarters of first-time entering 
ninth- graders now receive one of their top three high school choices. 2 

SSCs do not screen students based on their academic backgrounds. Instead SSC priori- 
ties are based solely on students’ geographic proximity and whether they are “known” to the 
school. Most SSCs have two categories of geographic proximity: they give priority to students 
who live in the New York City borough where the SSC is located over students who live 
elsewhere. Within each of these geographic categories, SSCs give priority to students who are 
known to them over students who are not known to them. (Thus most SSCs have four “geo- 
graphic by known” priority cells.) Students can become known to an SSC in many ways, such 
as by contacting it, visiting it, or meeting with its representative at a high school fair. 

When an SSC priority cell is oversubscribed by students from an annual cohort of in- 
coming ninth-graders, HSAPS randomly determines who is and is not assigned to it, thereby 
creating the statistical equivalent of a lottery for that cell (and thus for the SSC). This paper 


'HSAPS has three rounds of student placement. The second round places the vast majority of New 
York City entering ninth-graders and is the basis for the present analysis. The first round is used to assign 
students who have applied to one of New York City’s eight specialized high schools, and the third round is 
used to assign any students who are not placed by the second round or who want to dispute their second- 
round placement. For more information about HSAPS, see Abdulkadiroglu, Pathak, and Roth (2005). 

2 “Since HSAPS was first implemented in the 2003-2004 school year, the assignment algorithm has 
been adjusted to improve it. By the 2007-2008 school year, the process placed almost 50 percent of its 
students in their first-choice school and 80 percent in one of their first three choices.” Quint, Smith, 
Unterman, and Moedano (2010). 

3 Some SSCs have three geographic priorities: (1) residents of a nearby catchment area, (2) other resi- 
dents of their borough, and (3) other residents of New York City. 
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refers to these as “SSC lotteries.” Consider the following intuitive description of this process. It 
begins with the HSAPS algorithm randomly determining the order in which it will assign 
students to high schools. The first students assigned are the most likely to receive their first 
choice because no high schools are filled to capacity when HSAPS assigns them. 4 As high 
schools start to fill up, however, their student priorities begin to take effect. For example, if the 
first-choice high school of a student who is currently being placed by HSAPS is filled by 
students with equal or higher priority, the current student is not assigned to that school. Instead, 
he is assigned to his next-ranked school with available space. This process continues until 
HSAPS assigns all incoming ninth-graders in its queue. 

Figure A.l illustrates this process for a hypothetical student. In the example, HSAPS 
does not assign the student to his first-choice school (which is not an SSC) because he was 
chosen after the school was filled by other students with the same or a higher priority. The 
student is not assigned to his second-choice school (an SSC) either, because the school was 
already filled by students with a higher priority for it or because it was filled by students with 
the same priority for it. 5 If the school was already filled by students with the same priority, the 
current student would have been in a lottery for that SSC (described below) and lost it. If the 
SSC were filled by students with a higher priority, the current student would not have been in a 
lottery. Instead, he would have been preempted by students with higher priority for that school. 
Assume for the present discussion that the student lost a lottery for this SSC and thus became a 
member of its control group. 

The student is finally assigned by HSAPS to his third-choice school, which is an SSC. 
If he were assigned to this school because it was not oversubscribed by students with the same 
or a higher priority for it, he would not have been in a lottery for it. If he were assigned to the 
school and it was oversubscribed by students with the same priority for it, he would have been 
in a lottery for this school. Assume for the present discussion that the student was in a lottery for 
this SSC and won it. Thus, in the example, the hypothetical student lost a lottery for an SSC that 
was his second-choice school and won a lottery for an SSC that was his third-choice school and 
was assigned to the latter SSC. Consequently, the student was a control group “crossover” for 
his first SSC lottery (which was for his second-choice school). 

To demonstrate how HSAPS creates a lottery for an SSC when it is oversubscribed, 
Figure A.2 illustrates HSAPS assignment for a hypothetical SSC that can accommodate 120 
incoming ninth-graders. HSAPS will attempt to assign to this SSC all students who list it as one 


4 This is the case as long as students are not subsequently ‘'‘bumped” from the school by students who 
are assigned later by HSAPS and have higher priority for that school. 

5 If the SSC were filled by students with a lower priority. HSAPS would assign the current student to the 
SSC and “bump” the student with lowest priority that was most recently assigned to it. The student that was 
bumped would then be assigned to his next-ranked school with available space. 
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New York City Small Schools of Choice 
Figure A.l 

HSAPS Assignment Process for a Hypothetical Student 



of their choices and who are not assigned to a more preferred choice. For example, students 
who list the SSC as their third choice and do not receive their first or second choice (like the 
hypothetical student in the example) are considered for assignment to the SSC. In contrast, 
students who list the SSC as their third choice but receive their first or second choice are not 
considered for assignment to the SSC. 

The hypothetical SSC has 360 students who list it as a choice and do not receive a more 
preferred choice. Hence the SSC has 360 “potential assignees.” Eighty of them are from Priority 
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New York City Small Schools of Choice 
Figure A.2 

HSAPS Assignment Process for a Hypothetical SSC 
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No 


Yes 

No 

Priority Cell 1 

Priority Cell 3 

80 applicants assigned 

50 applicants not assigned 

Priority Cell 2 

Priority Cell 4 

40 applicants assigned 


120 applicants not assigned 

70 applicants not assigned 


Total SSC capacity = 120 
Total applicants = 360 

= Students assigned to SSC 


= Students not assigned to SSC 
= Cell containing a lottery 



Cell 1 (they live in the school’s borough and are known to it), 160 are from Priority Cell 2 (they 
live in the school’s borough and are not known to it), 50 are from Priority Cell 3 (they do not 
live in the school’s borough and are known to it) and 70 are from Priority Cell 4 (they do not 
live in the school’s borough and are not known to it). Given its capacity for entering ninth - 
graders, the SSC in the example can accept all 80 students from Priority Cell 1 plus the first 40 
students assigned by HSAPS from Priority Cell 2. It cannot accept the last 120 students from 
Priority Cell 2 or any students from Priority Cells 3 and 4. 
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Because Priority Cell 2 is oversubscribed, the 160 students in this cell are effectively 
“lottery participants.” The first 40 of these participants randomly selected for school assignment 
by HSAPS “win” the lottery and are assigned to the SSC. The last 120 participants randomly 
selected by HSAPS for school assignment “lose” the lottery and are not assigned to the SSC. 
Within this cell, only the random order in which HSAPS selects students for school assignment 
determines who wins or loses the lottery. 

Thus, at this point in the process, students ’ rank-ordered preferences for the SSC do not 
influence whether they are assigned to it. For example, it does not matter whether one lottery 
participant listed the SSC as his first choice and another listed it as his twelfth choice. The only 
thing that determines which participants win the lottery and which lose it is the random order in 
which HSAPS assigns students (that is, “who gets there first”). 

Note that no lottery exists for Priority Cell 1 of this SSC because all of its potential as- 
signees are assigned to the SSC. Furthermore, no lottery exists for Priority Cells 3 and 4 
because all of their potential assignees are preempted by students with a higher priority for that 
SSC. 6 Because only one cell has a lottery for a given SSC in a given year, the winners and 
losers of each lottery are matched or “blocked” by their priority for that SSC. In this way each 
SSC lottery produces a naturally occurring randomized trial. 

Before proceeding further it is important to note one final point: many participants in 
the hypothetical SSC lottery in Figure A.2 were not involved in a lottery for a more preferred 
high school. Hence, they were not involved in a logically “prior” HSAPS lottery. 7 The present 
lottery is thus their first HSAPS lottery. However, some participants in the current lottery (like 
the hypothetical student in his second lottery in Figure A.1) might have lost an HSAPS lottery 
for a more preferred school. These students did participate in a logically prior HSAPS lottery. 
Because outcomes of prior lotteries are determined “ before ” outcomes of the present lottery, 
whether or not students were in a prior HSAPS lottery is an exogenous “baseline characteristic” 
of current lottery participants, which can be used to produce two exogenously defined sub- 
groups of current lottery participants — those that were in a prior HSAPS lottery and those that 
were not. As described below, this fact plays an important role in the construction of the sample 
for the present report. 


6 HSAPS determines the geographic priority of each student for each school based on students’ address- 
es and schools’ priority categories. 

7 HSAPS also creates lotteries for some schools that are not SSCs. 
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How SSC Lotteries Produced the Present Sample 

Figure A.3 illustrates how SSC lotteries created the present sample. This sample represents the 
study’s first three annual cohorts of first-time ninth-graders who entered high school in the fall 
of 2005, 2006, or 2007 and: 

1. Who are in their first HSAPS lottery: 8 Including only students who are in 
their first HSAPS lottery produces a single sample point per student. 9 If a 
student loses her first lottery and is in another lottery for a less preferred 
SSC, and then is enrolled in an SSC because she wins this later lottery (like 
the hypothetical student in Figure A.l), she is a control group crossover for 
her first lottery. This fact is taken into account when estimating the effects of 
enrolling in an SSC (as described below). Any characteristic of a student who 
has not been in an HSAPS lottery that is determined before her current SSC 
lottery (like the schools she listed as more preferred choices than the SSC for 
the current lottery) is exogenous to the outcome of the current lottery and 
therefore is randomized by that lottery. 

2. Who are not from private or parochial schools: First SSC lottery partici- 
pants who are from private or parochial schools were excluded because the 
data available to MDRC researchers does not reliably track these students 
from their HSAPS participation into NYCDOE high schools. 10 Excluding 


8 The first two reports for the present study (Bloom, Levy Thompson, and Unterman, 2010; Bloom and Un- 
terman, 2012) are based on a sample that includes a separate sample point for each SSC lottery that a student is in. 
Thus for each SSC lottery there were sample points for participants who had and had not been in a prior HSAPS 
lottery. Appendix A of Bloom, Levy Thompson, and Unterman (2010) describes how it is possible to account for 
the clustering produced by these multiple sample points per student when estimating standard errors. The 
appendix also describes a theoretical threat to randomization for participants in a current SSC lottery who have 
been in a previous HSAPS lottery. Although the appendix presents extensive empirical evidence that this threat is 
very unlikely to be real, the issue increased the complexity of the analysis, created confusion among readers, and 
caused concerns about potential bias. The present analysis eliminates altogether this threat, and the confusion and 
concern it elicits, by only creating a sample point for each student’s first SSC lottery. Changing the sample 
structure did not affect the study’s findings appreciably. Both the new and the original sample structures produced 
estimates indicating that SSCs substantially increased high school graduation rates for the study sample overall 
and for almost all of its student subgroups. For example, findings for the new sample structure indicate that 
enrolling in an SSC increased the average four-year graduation rate for students in the study’s first three cohorts 
by 9.5 percentage points. The corresponding estimate for the original sample structure is 10.0 percentage points. 

9 Students who were in a prior lottery for a school that was not an SSC are also excluded from the pre- 
sent sample. This is done because the threat to randomization that is eliminated by dropping students from a 
current lottery who have been in a prior lottery occurs regardless of the type of school for which the prior 
lottery was conducted. 

1 ’Often, when private and parochial students participate in HSAPS they are given a unique identifier 
that exists solely for the purposes of the high school application process. In many instances this identifier is 
different from the NYCDOE identifier the students receive if they actually enroll in an NYCDOE high 
school the following school year. Thus, MDRC researchers do not have the ability to track these students 
from HSAPS participation to NY CDOE high school enrollment. 
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New York City Small Schools of Choice 
Figure A.3 

Construction of the Analysis Sample 



NOTES: 

a SSC lottery participants are first-time entering ninth-graders who were in their first HSAPS lottery. 
b lncomplete lotteries are those that are missing a treatment group or a control group. 

c Zero-compliance lotteries are those for which the percentage of winners who ever enrolled in an SSC equals the 
percentage of control group members who did so. 
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HSAPS participants who are from private or parochial schools provides an 
exogenous way to eliminate this problem because the nature of the prior 
school attended by sample members was determined before the outcomes of 
any SSC lotteries were determined. 

3. Who are not missing data on scores for both of their New York State 
eighth-grade tests (reading and math): First SSC lottery participants in this 
category were excluded because they probably did not live in New York 
State when they applied to HSAPS and only 3.0 percent of them could be 
identified as having enrolled the next year in an NYCDOE high school. This 
is probably because the identifiers needed to do so do not exist for most of 
the students who are not counted as having enrolled. Because whether or not 
students are missing both of their eighth-grade state tests was determined be- 
fore the outcomes of SSC lotteries were determined, this factor is an exoge- 
nous baseline characteristic of lottery participants. Thus excluding students 
who were missing this information is an exogenous exclusion. 

4. Whose lottery is complete (that is, still has both a treatment group and a 
control group) after students in the preceding categories are excluded from 
the sample. When students in the preceding two groups were excluded, 30 
lotteries (with 1,374 students) no longer had any treatment group members or 
no longer had any control group members. 11 Hence, they could not be used 
for the present analysis. 

5. Whose lottery is not for an SSC that did not open “as planned.” SSCs 
that did not open as planned were opened in a different building or a later 
year than was intended at the time of the lottery. Because students in these 
lotteries were assigned to high schools based on factors other than their lot- 
tery results, follow-up data for them does not provide rigorous information 
about SSC effects. 12 

6. Whose lottery has nonzero compliance (that is, the percentage of lottery 
winners who enrolled in an SSC differs from that of control group members 
who did so). For six SSC lotteries (with 110 students) the percentage of lot- 
tery winners who subsequently enrolled in an SSC equals the percentage of 

n Each of these lotteries had only one or two treatment group members or one or two control group 
members before the sample exclusions were invoked. 

'"Technically these schools did not open the following fall with their HSAPS Round Two school code 
or HSAPS Round Two assignment process students. In these instances the school opening was delayed a 
year, the school location was moved, or the school admitted students using the HSAPS Round 
Three/summer process. 
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control group members who did so. 13 Because these lotteries have no SSC 
treatment contrast they provide no infonnation about the effect of enrolling 
in an SSC. 

The preceding criteria resulted in a “final lottery sample” of 14,969 students from 199 
lotteries for 84 SSCs. Table A.l demonstrates that mean values for a broad range of relevant 
baseline characteristics are virtually identical for sample members who were lottery winners 
(treatment group members) and sample members who were lottery losers (control group 
members). These comparisons account for the fact that randomization was “blocked” by lottery. 
To do so, the treatment and control group difference in the mean value of a given student 
baseline characteristic (X) was estimated as the value of fi x from the following regression. 

X i = l J j=1 a j -Ij i +(3 x -T i + v i (A.l) 

where: 

X t — the value of baseline characteristic X for student i, 

Iji = one if lottery j was the first SSC lottery for student i and zero otherwise, 

7) = one if student i won his first SSC lottery (was a treatment group member for it) 
and zero if he lost his first SSC lottery (was a control group member for it), 
and 

Vi = a random error that is independently and identically distributed across students 
within lotteries. 

The first column in Table A.l reports the sample mean of each baseline characteristic 
for lottery winners (treatment group members). The third column reports estimates of the 
treatment and control group difference in means for each baseline characteristic, controlling for 
students’ lottery (the estimated value of fS x ). The fourth column reports the level of statistical 
significance (p-value) for each estimated difference of means. The second column reports the 
inferred mean of each baseline characteristic for control group members. This value was 
obtained by subtracting the estimated difference of means for treatment and control group 
members from the corresponding mean for treatment group members. All estimates of the 
difference between mean baseline characteristics for treatment and control group members are 
small and none is statistically significant. 

The final step in constructing the “four-year follow-up sample” for the present analysis 
was to account for student attrition. Figure A.3 illustrates that this attrition represents 2,553 


l3 These lotteries were very small, with total samples ranging from 4 to 24 students and control groups 
ranging from only 1 to 3 students. All lottery winners and control group members enrolled in an SSC. 
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New York City Small Schools of Choice 
Table A.l 

Baseline Characteristics of First-Time SSC Lottery Participants: 
Final Lottery Sample, Cohorts 1-3 



SSC 

Control 


P-Value for 


Lottery 

Group 

Estimated 

Estimated 

Characteristic (%) 

Winners 

Members 

Difference 

Difference 

Race/ethnicity 





Hispanic 

44.7 

46.7 

-2.0 

0.074 

Black 

45.1 

43.6 

1.6 

0.114 

American Indian 

0.5 

0.5 

0.0 

0.862 

White 

4.7 

4.5 

0.2 

0.682 

Asian 

3.5 

3.4 

0.1 

0.740 

Male 

47.8 

46.5 

1.3 

0.201 

Eligible for free/reduced-price lunch 

83.5 

84.0 

-0.6 

0.537 

English language learner 

6.9 

7.0 

-0.1 

0.874 

Special education 2 

5.4 

5.8 

-0.4 

0.476 

Overage for 8th grade* 3 

18.4 

19.3 

-0.9 

0.281 

8th-grade reading proficiency 2 





Did not meet standards (level 1) 

7.4 

6.7 

0.7 

0.183 

Partially met standards (level 2) 

62.4 

63.5 

-1.1 

0.283 

Fully met standards (level 3) 

29.3 

29.1 

0.2 

0.842 

Met standards with distinction (level 4) 

1.0 

0.8 

0.2 

0.396 

8th-grade math proficiency 2 





Did not meet standards (level 1) 

20.2 

19.9 

0.3 

0.763 

Partially met standards (level 2) 

47.6 

46.8 

0.8 

0.444 

Fully met standards (level 3) 

30.5 

31.6 

-1.1 

0.239 

Met standards with distinction (level 4) 

1.7 

1.7 

0.0 

0.916 

Missing 8th-grade pretests 





Math proficiency 

0.8 

1.2 

-0.4 

0.097 

Reading proficiency 

3.3 

3.3 

0.0 

0.998 

Sample size (total = 14,969) 

6,230 

8,739 



Number of lotteries (total = 199) 





Number of SSCs (total = 84) 





(continued) 
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Table A.l (continued) 


SOURCES: MDRC’s calculations use High School Application Processing System and New York 
City Department of Education (NYCDOE) state test data for eighth-graders in 2004-2005 (cohort 
1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3). 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the difference between SSC lottery winners and control group members are obtained from a 
regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. The 
coefficient on the latter indicator variable equals the difference in the mean baseline characteristic 
for lottery winners and control group members. The value for control group members equals the 
corresponding value for SSC lottery winners minus the estimated difference between lottery 
winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in 
the table. The resulting chi-square value is not statistically significant (p-value = 0.374 ). 

a This sample includes special education students who can be taught in a regular classroom 
setting. Special education students classified by the NYCDOE as requiring collaborative team 
teaching services or self-contained classes are not part of the sample. 

b Lottery participants are classified as “overage for eighth grade” if they were 14 years old or 
older on September 1 of their eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade 
level on state math and reading exams. Due to a small percentage of missing test scores, the 
percentages for levels 1-4 may not sum to 100 percent. 


students who are missing four-year follow-up data and 286 students from seven lotteries that 
became “incomplete” when students with missing follow-up data were omitted. 14 

The resulting sample contains 12,130 students from 192 lotteries for 84 SSCs, which 
represents an overall student attrition rate of 19.0 percent. An estimate of the treatment and 
control group difference in student attrition rates that accounts for the blocking of randomiza- 
tion by lottery (using a model like Equation A. 1 with a binary attrition indicator as its dependent 
variable) indicates that this difference is only 0.53 percentage points and is not statistically 
significant (p-value = 0.481). Thus winning or losing an SSC lottery does not affect the rate of 
student attrition. In addition findings in Table A.2 (obtained by estimating Equation A.l from 
data for the four-year follow-up sample) indicate that there is virtually no treatment and control 
group difference in baseline characteristics for the four-year follow-up sample. This is strong 
evidence that student attrition does not affect the internal validity of the present findings. 


14 Six of these lotteries had only one to three treatment group members or one to three control group 
members before the sample exclusions were invoked. One of them had 42 control group members and 91 
treatment group members. 
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New York City Small Schools of Choice 
Table A.2 


Baseline Characteristics of First-Time SSC Lottery Participants: 
Four-Year Follow-Up Sample, Cohorts 1-3 



SSC 

Control 


P- Value for 


Lottery 

Group 

Estimated 

Estimated 

Characteristic (%) 

Winners 

Members 

Difference 

Difference 

Race/ethnicity 





Hispanic 

43.7 

45.3 

-1.6 

0.148 

Black 

46.0 

44.9 

1.1 

0.314 

American Indian 

0.4 

0.5 

0.0 

0.811 

White 

5.0 

5.1 

-0.1 

0.809 

Asian 

3.6 

3.1 

0.5 

0.257 

Male 

47.1 

45.6 

1.5 

0.182 

Eligible for free/reduced-price lunch 

83.4 

83.2 

0.3 

0.796 

English language learner 

6.1 

5.9 

0.2 

0.762 

Special education 1 ' 

5.4 

5.5 

-0.1 

0.889 

Overage for 8th grade* 3 

16.7 

17.1 

-0.4 

0.686 

8th-grade reading proficiency 1 " 





Did not meet standards (level 1) 

6.6 

5.9 

0.7 

0.261 

Partially met standards (level 2) 

61.8 

63.2 

-1.4 

0.208 

Fully met standards (level 3) 

30.5 

30.0 

0.5 

0.642 

Met standards with distinction (level 4) 

1.1 

0.8 

0.3 

0.229 

8th-grade math proficiency" 





Did not meet standards (level 1) 

19.1 

18.5 

0.6 

0.524 

Partially met standards (level 2) 

47.5 

47.2 

0.2 

0.834 

Fully met standards (level 3) 

31.6 

32.4 

-0.8 

0.451 

Met standards with distinction (level 4) 

1.9 

1.9 

-0.1 

0.865 

Missing 8th-grade pretests 





Math proficiency 

0.7 

0.8 

-0.1 

0.543 

Reading proficiency 

3.0 

2.9 

0.1 

0.815 

Sample size (total = 12,130) 

5,020 

7,110 



Number of lotteries (total = 192) 





Number of SSCs (total = 84) 





(continued) 
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Table A.2 (continued) 

SOURCES: MDRC’s calculations use High School Application Processing System data and New 
York City Department of Education (NYCDOE) state test data for eighth-graders in 2004-2005 
(cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus data from NYCDOE enrollment 
files for the 2005-2006 to 2010-2011 school years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the difference between SSC lottery winners and control group members are obtained from a 
regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. The 
coefficient on the latter indicator variable equals the difference in the mean baseline characteristic 
for lottery winners and control group members. The value for control group members equals the 
corresponding value for SSC lottery winners minus the estimated difference between lottery 
winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in 
the table. The resulting chi-square value is not statistically significant (p-value = 0.430). 

a This sample includes special education students who can be taught in a regular classroom 
setting. Special education students classified by the NYCDOE as requiring collaborative team 
teaching services or self-contained classes are not part of the sample. 

b Lottery participants are classified as “overage for eighth grade’’ if they were 14 years old or 
older on September 1 of their eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade 
level on state math and reading exams. Due to a small percentage of missing test scores, the 
percentages for levels 1-4 may not sum to 100 percent. 


How Data for the Present Analysis Were Obtained 

The primary sources of data for this report are information from the High School Application 
Processing System (HSAPS) and from NYCDOE school records for individual students. This 
infonnation was obtained from the NYCDOE. In addition, publicly available data on school 
characteristics were obtained from New York State’s School Report Cards and the U.S. De- 
partment of Education’s Common Core of Data. 15 

HSAPS data were used to identify students who participated in an SSC lottery, to de- 
termine the school to which they were assigned and the school in which they enrolled, to 
describe their background characteristics, and to compare background characteristics for SSC 
lottery winners and control-group members. These data include students’ rank-ordered lists of 

15 Appendix E of Bloom, Levy Thompson, and Unterman (2010) describes these data. 
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preferred high schools, their baseline characteristics, and SSCs’ priority categories for incoming 
students. 

Students’ school records data were used to construct follow-up measures of academic 
progress, which are the basis for estimates of SSC effects on student outcomes. This infor- 
mation includes enrollment, attendance, and course credits earned, plus state test scores and 
results of state Regents examinations. Middle-school data on eighth-grade standardized test 
scores in reading and math were obtained for baseline comparisons of SSC lottery winners and 
control-group members. 


How SSC Effects Were Estimated 

Although winning an SSC lottery is the same as being assigned by HSAPS to an SSC, losing an 
SSC lottery is not the same as not being assigned by HSAPS to an SSC. This is because a 
student who loses one SSC lottery can be assigned by HSAPS to an SSC that was lower on his 
rank-ordered list of school preferences. Thus a simple comparison of outcomes for SSC lottery 
winners and losers does not represent the causal effect of being assigned (or not) to an SSC. 
Indeed the result of such a comparison does not have a useful causal interpretation. Nonetheless, 
to provide a point of comparison, Appendix B presents estimates of the effect of winning a 
student’s first SSC lottery on graduation and college readiness. This table is the direct counter- 
part to Table 1 in the main text of the present report. 

This report focuses its analyses on estimates of the causal effect of enrolling in an SSC, 
using an approach that is widely employed for randomized trials to estimate effects of receiving 
a treatment. The approach uses random assigmnent to treatment or control status (winning or 
losing one’s first SSC lottery) as an instrumental variable or instrument, to predict SSC enroll- 
ment, which in turn is used to estimate the causal effect of enrolling in an SSC. 16 

This approach produces an estimate of the average effect of enrolling in an SSC for stu- 
dents who did so because they won their first SSC lottery. This type of causal effect is typically 
referred to as a local average treatment effect and the individuals to which this effect applies are 
often referred to as “compliers” (because they comply with their assigned treatment). 17 But 
compliance is an ambiguous concept for students who lose one SSC lottery, win another, and 
consequently enroll in an SSC. Such students are noncompliers for the first lottery and potential 
compliers for the second. To avoid this ambiguity, this report uses the term “target SSC enrol- 


16 For discussions of this approach to analyzing multisite randomized trials see Gennetian, Morris, Bos, 
and Bloom (2005); Ludwig and Kling (2007); and Kling, Liebman, and Katz (2007). For analyzing 
randomized studies in general, see Angrist, Imbens, and Rubin (1996) and Angrist and Pischke (2009). For 
analyzing lottery-based studies in particular, see Abulkadiroglu et al. (201 1). 

17 Angrist, Imbens, and Rubin (1996). 
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lee” to designate students for whom SSC enrollment effects are estimated. This term was 
chosen because these students are the target of estimation. 

In a multiblock randomized trial like the present analysis (with SSC lotteries as blocks), 
researchers typically specify a separate instrument for each block. These instruments are created 
by interacting a binary indicator for treatment assignment (T L ) with a binary indicator for each 
block (l j L ). The present instruments are valid because they are randomized and strong because 
they are highly predictive of SSC enrollment. 18 To increase precision, students’ scores on their 
eighth-grade New York State tests of math and reading were included as baseline covariates. 
Two-stage least squares was then used to estimate a model like that described by Equations A.2 
and A.3. 

First Stage: SSC Enrollment as a Function of First SSC Lottery Assignment 

= Zy= i Tlj ' Iji + £/= i Yj ' T i ' Iji + ' S Mi + Sr ' S Ri + w i ( A - 2 ) 

where: 


E t = one if student i enrolled in an SSC during the four-year follow-up period and zero 
otherwise, 19 

Iji = one if lottery j was the first SSC lottery for student i and zero otherwise, 

T t = one if student i won his first SSC lottery and zero otherwise, 

S Mi and S Ri = student i’s eighth-grade scores on New York State tests of math and 
reading, 20 and 

Wi = a random error that is distributed independently and identically across students 
within lotteries. 

Second Stage: High School Outcome as a Function of Predicted SSC 
Enrollment 

Y i ~ £y= i a j ' Iji + 8 ' Ei + 4 >m ' S Mi + 0R ' S Ri + e i ( A -3) 


where: 


18 See Bloom, Zhu, and Unlu (2010) or Angrist and Pischke (2009) for a discussion of instrument 
strength and finite sample bias. All reported results have a first-stage F statistic equal to 10 or more, as 
recommended by Stock and Yogo (2005) to minimize finite sample bias. 

’’Although we define enrolling in an SSC as enrolling in any SSC, 87.6 percent of the first SSC lottery 
winners who enrolled in an SSC enrolled in the SSC for that lottery’. Thus, enrolling in an SSC almost 
always means enrolling in the SSC for the lottery that a student won. 

Approximately 0.8 percent of sample members were missing their eighth-grade math pretest scores 
and 2.4 percent were missing their eighth-grade reading pretest scores. These missing values were imputed 
using a single model-based imputation that was based on all available baseline and follow-up data for 
sample members. The statistical software routine SAS PROC MI was used for this purpose. 
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Y l = the outcome for student i, 

Ei = the predicted value of SSC enrollment for student i from the estimated first-stage 
equation, and 

e ( = a random error that is clustered by the first school that students entered after their 
lottery. 21 

The estimated value of S is a consistent estimate of the average effect of enrolling in an 
SSC for target SSC enrollees (sample members who were induced to enroll in an SSC by 
winning their first SSC lottery). 22 This is a local average treatment effect or LATE. 23 


How Estimates of SSC Effects Are Reported 

Because estimates of SSC effects are for target SSC enrollees (students who comply with their 
first lottery assignment) it is important to understand not only how these effects were estimated 
but also how they are reported and how the additional information with which they are reported 
was obtained and should be interpreted. As a basis for doing so consider the findings in the first 
row of Table A.3, which represent the estimated effect of enrolling in an SSC on the four-year 
high school graduation rate for target SSC enrollees in the four-year follow-up sample. This 
result, which is the present report’s central finding, indicates that enrolling in an SSC increased 
four-year graduation rates by 9.5 percentage points (to 70.4 percent for target SSC enrollees 
from 60.9 percent for their control group counterparts). Because this finding is based only on 
data for members of the four-year follow-up sample it involves no imputation of missing data. 


The estimated SSC effect in the third column of the table is that produced by estimating 
S from the two-stage least squares model represented by Equations A.2 and A.3 and the 
statistical significance level of this estimate (its p-value) in the fourth column of the table also 
comes directly from estimating this model. The size of the sample upon which these findings 
are based, 12,130 students, represents the size of the four-year follow-up sample. 

To provide a contextual “anchor” for estimates of SSC effects on the high school grad- 
uation rate for target SSC enrollees and to provide benchmarks for gauging the relative magni- 
tudes of these estimates, they are reported alongside corresponding estimates of mean gradua- 
tion rates for target SSC enrollees and their control group counterparts (in columns one and two 
of the table, respectively). Versions of this approach have been used in the past to report 
estimates of intervention effects from randomized trials. These findings use as their contextual 


_ 1 This cluster adjustment has very little influence on estimated standard errors. 

22 Estimated second -stage standard errors are adjusted to account for the uncertainty in predicted values 
of Ei. 

23 Angrist, Imbens, and Rubin (1996). 
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anchor observed mean outcomes for treatment group members. 24 This step of the analysis is 
typically straightforward because treatment group members can be identified individually and 
their mean outcomes can be computed directly. 

However the present study reports estimates of mean outcomes for a subgroup of treat- 
ment group members who cannot be identified individually — target SSC enrollees. Hence 

New York City Small Schools of Choice 
Table A.3 

Estimated Effects of SSC Enrollment on Four-Year Graduation Rates 
Accounting for Student Attrition: Cohorts 1-3 


Graduated from High School (%) 

Target 

SSC 

Enrollees 

Control 

Group 

Counterparts 

Estimated 

Effect 

P-Value for 
Estimated 
Effect 

Sample 

Size 

Four-vear follow-up sample 3 






No imputed outcomes 

70.4 

60.9 

9.5 ** 

0.000 

12,130 

Final SSC lottery sample b 






Model-based imputed outcomes 

64.6 

56.1 

8.5 ” 

0.000 

14,969 

Imputed outcomes set to zero 

59.3 

50.7 

8.5 ** 

0.000 

14,969 


SOURCES: MDRC’s calculations use High School Application Processing System data for 
eighth-graders in school year 2004-2005 (cohort 1) to 2006-2007 (cohort 3), plus data from 
New York City Department of Education attendance, course credits, Regents exam, 
transactional, and enrollment files for the 2005-2006 to 2010-2011 school years. 

NOTES: Appendix A describes how values in the table were estimated. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

a The four-year follow-up sample contains all 12,130 students in the final SSC lottery 
sample that have four-year follow-up data. 

b The final SSC lottery sample contains all 14,696 students in eligible SSC lotteries. 


these estimates must be obtained indirectly. This is made possible by the statistical properties of 
randomization plus two plausible assumptions. Figure A.4 presents the conceptual model that 
underlies this process. The model portrays two main subgroups of SSC lottery winners: those 
who enroll in an SSC (a large majority) and those who do not enroll in an SSC (a small 


24 See Black et al. (2009); Corrin et al. (2008). 
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minority who become “no-shows”). 25 The model also portrays two main subgroups of SSC 
control group members: those who do not enroll in an SSC (a large majority) and those who 
do enroll in an SSC (a small minority who become “crossovers”). 26 


New York City Small Schools of Choice 
Figure A.4 

Model of SSC Enrollment Among Lottery Winners and Control Group Members 


SSC Lottery Control Group 

Winners Members 


No-shows 

(Do not enroll in SSC) 


No-show counterparts 

(Do not enroll in SSC) 

Target SSC enrollees 

(Enroll in SSC) 


Control group counterparts 
for target SSC enrollees 
(Do not enroll in SSC) 

Crossover counterparts 

(Enroll in SSC) 


Crossovers 
(Enroll in SSC) 


Assume (as seems plausible) that students are more likely to enroll in an SSC if they 
win an SSC lottery than if they do not. Given this assumption (often referred to as “mono- 
tonicity” or the absence of “defiers”), 27 there are two more subgroups in the model: “no-show 
counterparts” among control group members (control group members who do not enroll in an 
SSC and would not have done so even if they had won an SSC lottery) and crossover coun- 
terparts among SSC lottery winners (lottery winners who enroll in an SSC and would also 


25 See Bloom (1984) for a definition and discussion of no-shows. 

26 See Bloom et al. (1997) for a definition and discussion of crossovers. 
27 As defined and discussed by Angrist, Imbens, and Rubin (1996). 
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have done so if they had been randomized to the control group). 28 Lottery winners who are 
neither no-shows nor crossover counterparts are target SSC lottery enrollees. This is the 
subgroup of students for whom SSC effects were estimated, and it has a counterpart subgroup 
among control group members. 

Randomization ensures that in expectation : (1) the proportion of SSC lottery winners 
who are no-shows ( P NS ) equals the proportion of control group members who are no-show 
counterparts and (2) the proportion of control group members who are crossovers ( P co ) equals 
the proportion of SSC lottery winners who are crossover counterparts. Hence the proportion 
of target SSC enrollees among SSC lottery winners equals the proportion of control group 
members who are their counterparts. 

Now assume that winning an SSC lottery in itself has no appreciable direct effect on 
future student academic performance and that only by causing students to enroll in an SSC 
can winning a lottery affect these outcomes (which often is referred to as the “exclusion” 
restriction). 29 This assumption is highly plausible given that students do not even know they 
are in lotteries. If the assumption holds, randomization ensures that in expectation, mean 
outcomes for crossovers in the control group (Y c0 ) equal those for crossover counterparts 
among SSC lottery winners. 


Now note that the mean value of an outcome for all SSC lottery winners who enroll in 
an SSC (Y lwe ) is a weighted average of mean outcomes for target SSC enrollees (Y tarE ) and 
crossover counterparts with weights equal to the relative size of each group. Then note that the 
observed mean outcome for crossovers ( Y C) ) is an unbiased estimate of the mean outcome for 
crossover counterparts. Together, these facts imply that: 


Ylwe 


1_ p NS~ p CO y 

1-Pss ' tarE 


p co 

-1 -Pns- 


' CO 


(A. 4) 


Solving Equation A.4 for Y tarE yields: 
i -Pns 


YtarE 


Li-Pjvs-Pco-I 


Ylwe ~ 


p co 
L ^-Pns-Pco- 


l co 


(A. 5) 


In this way, the implied value of Y tarE can be estimated from observed values of P NS , 
P C o , Y lwe , and Y c0 . This is how the findings in the first column of Table A.3 were obtained 
(along with their counterparts elsewhere in this report). 


28 Angrist, Imbens, and Rubin (1996) refer to no-shows and their control-group counterparts as “never 
takers” and to crossovers and their treatment-group counterparts as “always takers.” 

29 Angrist, Imbens, and Rubin (1996). 
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The findings in the second column of the table, which are estimates of the mean four- 
year graduation rate for control group counterparts, are then obtained by subtracting the esti- 
mated effect of enrolling in an SSC (in the third column) from the estimated graduation rate for 
target SSC enrollees (in the first column). 


How Sample Attrition Was Addressed 

Recall that the present four-year follow-up sample is missing 19 percent of the students from 
the final lottery sample. Because (as noted above) this rate of attrition is virtually the same for 
treatment and control group members and because (as demonstrated by Table A.2) treatment 
and control group members in the four-year follow-up sample are virtually identical in temis of 
relevant observed baseline characteristics, the internal validity of resulting estimates of SSC 
effects should be strong. 

However, it is useful to consider further the likely effect of student attrition on both the 
internal and external validity of these findings by examining their sensitivity to a range of 
plausible assumptions about future high school outcomes for students lost to attrition. With this 
in mind, recall that present attrition involves 2,553 students who are missing four-year follow- 
up data and 286 students from 7 lotteries that became “incomplete” (lost their control group or 
treatment group) when students with missing follow-up data are omitted. Because the second 
type of attrition involves losing whole lotteries, it is extremely unlikely to affect the internal 
validity of findings for the remaining sample. However students with missing follow-up data 
could, in theory, reduce the internal validity of study findings if they caused treatment and 
control group members in the remaining sample to differ in unobserved ways that are related to 
their counterfactual outcomes. Furthermore, the loss of both types of students could, in theory, 
reduce the external validity of study findings. 

Findings in Table A.3 demonstrate that this attrition has very little influence on esti- 
mates of SSC effects on high school graduation rates, although it does influence estimates of the 
underlying graduation rates themselves. Consider the second and third rows in Table A.3, which 
report findings for the final lottery sample of 14,969 students, using two different approaches to 
imputing missing follow-up data. The first approach used SAS PROC MI to perform a single 
model-based imputation from all existing baseline and follow-up data. These findings indicate 
that enrolling in an SSC increased high school graduation rates by 8.5 percentage points (to 64.6 
percent for target SSC enrollees from 56.1 percent for their control group counterparts). Thus 
both the estimated effect of SSCs and the corresponding graduation rates for target SSC 
enrollees and their control group counterparts declined with the addition of imputed values for 
missing outcomes. This is because students with missing four-year outcome data were much 
more likely than others to have entered high school overage for grade and much less likely than 
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others to have been on track toward graduation in ninth grade. 30 Consequently, their graduation 
prospects were much lower than those of students who are not missing four-year follow-up data. 

The third row in the table presents findings for the final lottery sample which assume that 
no students with missing follow-up data graduated from high school in four years. Although this 
conservative assumption produces even lower estimates of graduation rates, its estimate of the 
SSC effect is the same as that for the model-based imputation, 8.5 percentage points. 31 


30 On average, 31.3 percent of the students who were missing four-year outcome data entered high 
school overage for grade, while only 18.7 percent of other students in the four-year follow-up sample were 
overage for grade at this time. This is a difference of 12.6 percentage points. After one year of high school, 
28.9 percent of the students who were missing four-year follow-up data but had follow-up data for one year 
were on track to graduate high school in four years, while 55.1 percent of other students in the four-year 
follow-up sample were on track to graduate in four years after one year of high school. This is a difference 
of 26.2 percentage points. 

31 lt is not plausible that the graduation rate for students lost to attrition who were lottery winners is low- 
er than that for students lost to attrition who were control group members because SSCs increased early 
progress toward graduation for the 94 percent of students lost to four-year attrition with available data on 
this early outcome. Specifically, enrolling in an SSC increased the likelihood of being on track toward 
graduation in ninth grade for students lost to attrition by 2.4 percentage points (to 29.0 percent for target 
SSC enrollees from 26.6 for their control group counterparts). This positive finding was not statistically 
significant, however (p-value = 0.436). 


51 




Appendix B 

Estimated Effects of Winning a Student’s First SSC 

Lottery 




New York City Small Schools of Choice 
Table B.l 

Estimated Effects of Winning a Student’s First SSC Lottery on 
Four-Year High School Graduation and College Readiness: 

Cohorts 1-3 


Outcome (%) 

Lottery 
W inners 

Control 

Group 

Counterparts 

Estimated 

Effect 

P- Value for 
Estimated 
Effect 

Graduation 

Graduated from high school 

69.6 

63.6 

6.0 ** 

0.000 

Local diploma granted 

17.1 

15.5 

1.6 

0.061 

Regents diploma granted 

44.9 

41.3 

3.5 ** 

0.002 

Advanced Regents diploma granted 

7.5 

6.8 

0.8 

0.172 

College readiness 

English Regents exam score of 75 or above 

39.5 

35.4 

4. 1 ** 

0.000 

Math A Regents exam score of 75 or above 

24.3 

24.6 

-0.3 

0.790 

Sample size 

5,020 

7,110 




SOURCES: MDRC’s calculations use High School Application Processing System data for eighth- 
graders in 2004-2005 (cohort 1), 2005-2006 (cohort 2), and 2006-2007 (cohort 3), plus data from the 
New York City Department of Education’s fdes on student attendance, course credits. Regents 
examination scores, administrative transactions, and enrollment for the 2005-2006 to 2010-2011 
school years. 

NOTES: The value of each outcome for SSC lottery winners is the simple mean for all lottery 
winners. Values for the estimated effect on each outcome measure of winning a student’s first lottery 
were obtained from a regression of the outcome measure on a series of indicator variables that 
identify each lottery, plus an indicator variable that equals 1 for lottery winners and 0 for lottery 
losers, plus students scores on their eighth-grade state tests in reading and math. The estimated 
coefficient on the lottery winner/loser indicator variable equals the estimated effect on the outcome 
of winning a student’s first SSC lottery. The value for control group members equals the outcome 
value for SSC lottery winners minus the estimated effect of winning a student’s first SSC lottery. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated 
as ** = 1 percent and * = 5 percent. 
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Appendix C 


2008 Requirements for Proposals to Create New SSCs 

Specified by the New York City Department of 

Education 1 


'New York City Department of Education (2008). 




The Office of Portfolio Development will evaluate new school applications using research- 
based evidence of the following Elements of Effective Schools : 

1. Strong leadership and a mission that teachers, administrators, and students 
know and support. 

2. A structure, including elements such as reduced teacher load that ensures that 
students will be known well by their teachers and other school staff. 

3. A small team of qualified teachers responsible for a manageable number of 
students for at least a full school year that has the autonomy necessary to de- 
termine what students learn and how and what they need to make regular 
progress towards graduation. 

4. High expectations for all students and a standards-based, academically rigor- 
ous curriculum that connects what students leam with college and career 
goals. 

5. Perfonnance-based assessments for students and teachers and a culture of 
continuous improvement and accountability for student success. 

6. A structure that fosters the development of authentic, sustained, caring, and 
respectful relationships between teachers and students and among staff 
members. Advisories of 10-15 students are one strategy to achieve this goal. 

7. A school schedule that includes longer instructional blocks that promote in- 
terdisciplinary work, teacher collaboration, and reduced student loads. This 
schedule should be coupled with collaborative team planning and profession- 
al development time within the regular school schedule so that teachers can 
form a professional community. 

8. A well-defined plan to service the learning needs of the full range of students 
in the community, including special education students and English Lan- 
guage Learners. 

9. Connections between what students leam in school to their lives and com- 
munities through internships, mentoring experiences, and service learning 
opportunities. 

10. Partnerships with students, parents, and community organizations and institu- 
tions as key collaborators and stakeholders. 
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As part of Children First, new schools created in 2008 are designed specifically to meet 
the needs of under-served communities. Applications should clearly state how all students, 
including those who are performing below grade level, students entitled to special education 
services, and English language learners will be successful in this new school design. 
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Appendix D 

Documentation for Interviews 
and Focus Groups 




MDRC contracted with the Research Alliance for New York City Schools to conduct inter- 
views with principals and focus groups with teachers from a sample of SSCs that had strong 
evidence of effectiveness. MDRC selected the SSCs and Research Alliance researchers de- 
signed the protocols for this fieldwork, conducted the interviews and focus groups during the 
201 1-2012 school year, prepared a data set from the resulting findings, and conducted the initial 
data analyses. 

Because the goal of this fieldwork was to leam as much as possible from talking to 
principals and teachers at effective SSCs, MDRC chose the 30 SSCs that had the largest or most 
statistically significant positive estimates of effects on ninth-graders’ progress toward gradua- 
tion during the two most recent school years for which data were available at the time (2008- 
2009 and 2009-2010). These estimates were produced in the same way as those for MDRC’s 
previous SSC reports. 1 

After vigorous recruitment, the Research Alliance conducted interviews and focus 
groups at 25 of the targeted SSCs, which was the goal set for the recruitment effort. Principals at 
two SSCs declined to have their schools participate because of concerns about time limitations. 
Principals at three SSCs did not respond to the numerous attempts to recruit them, including in- 
person visits. 

It is important to note that the 30 targeted SSCs, and the resulting 25 SSCs in the field- 
work sample, do not necessarily represent all SSCs that were effective. Some SSCs did not have 
student lotteries for the years examined (the basis for estimating effectiveness) and others had 
lottery samples that were too small to provide strong evidence. This lack of evidence about 
effectiveness does not necessarily imply a lack of effectiveness. In addition, because estimates 
of SSC effects that guided the sample selection contain error (like any other estimates) it is 
possible that not all of the SSCs in the resulting fieldwork sample were effective. However, 
there is strong evidence that as a group, on average, these SSCs are effective. 


Fieldwork SCCs 

Table D.l describes key features of the 25 fieldwork SSCs. To help assess the likely generaliza- 
bility of the fieldwork findings, the table compares key features of the 25 fieldwork SSCs with 
those for the population of 1 10 SSCs that were created between the fall of 2002 and the fall of 


'Bloom, Levy Thompson, and Unterman (2010); Bloom and Unterman (2012). The steps in this process 
were as follows: (1) identify SSCs with impact estimates for both years (54), (2) estimate the average SSC impact 
for the two years combined, (3) sort these estimates by their magnitude, (4) select all 18 estimates that were 
statistically significant at the 0.100 level (18), (5) select the remaining 12 estimates that were not statistically 
significant in order of their magnitude (p-values for these estimates ranged from 0.101 to 0.359). 
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New York City Small Schools of Choice 
Table D.l 


Characteristics of the 25 Fieldwork SSCs, All SSCs, 
and All Other NYCDOE High Schools in 2010-2011 


Characteristic 

Fieldwork 

SSCs a 

(25) 

All 

SSCs b 

(110) 

All Other NYCDOE 
High Schools 0 
(320) 

Student characteristics (%) 

Race/ethnicity 




Hispanic 

50.0 

48.4 

42.4 

Black 

44.1 

44.5 

35.8 

White 

2.4 

3.3 

9.9 

Asian 

2.7 

3.3 

11.4 

Other 

0.8 

0.5 

0.5 

Male 

53.4 

50.9 

49.2 

Eligible for free/reduced-price lunch 

86.0 

78.2 

72.6 

Special education 

18.7 

17.6 

14.1 

English language learner 

9.1 

12.4 

13.8 

Overage for 8th grade d 

30.5 

28.6 

22.1 

Scored below grade level on 8th-grade reading test e 

77.4 

78.0 

65.0 

Scored below grade level on 8th-grade math test 

68.9 

70.9 

55.0 

Teacher characteristics (%) 

Teachers with less than 3 years of experience 

9.1 

9.2 

9.8 

Teachers with a master’s plus a certificate 

38.9 

34.2 

44.0 

Teachers teaching out of their certification 

11.1 

10.3 

10.7 

Teacher turnover rate 8 

23.2 

21.7 

20.4 

School characteristics 

Total school enrollment 

426 

389 

891 

Average lOth-grade class size* 1 

28 

27 

28 

Borough (%) 

Bronx 

54.2 

44.0 

21.6 

Brooklyn 

33.3 

33.9 

26.9 

Manhattan 

8.3 

14.7 

28.1 

Queens 

4.2 

6.4 

20.6 

Staten Island 

0.0 

0.9 

2.8 

(continued) 
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Table D.l (continued) 


Characteristic 

Fieldwork 

SSCs a 

(25) 

All 

SSCs b 

(110) 

All Other NYCDOE 

High Schools 0 
(320) 

School age as of 2011 (%) 
1-3 years 

0.0 

0.0 

0.0 

4 years 

0.0 

10.3 

2.4 

5 years 

0.0 

9.3 

5.6 

6 years 

12.5 

17.8 

4.4 

7 years 

45.8 

41.1 

6.3 

8 years 

41.7 

15.9 

6.3 

9 or more years 

0.0 

5.6 

75.0 


SOURCES: Data on special education status and overage for grade status came from the 2010- 
2011 NY CDOE Progress Report. Data on race, free and reduced-price lunch status, English 
language learner status, average tenth-grade class size, teachers with less than three years of 
experience, teachers with a master’s plus a certificate, teachers teaching out of their certification, 
teacher turnover rate, and total school enrollment came from the 2010-2011 New York State 
Report Card. Data on student eighth-grade reading and math tests scores were provided by the 
NYCDOE. 

NOTES: 

a The 25 Small Schools of Choice (SSCs) where interviews with principals were conducted 
during the 2010-2011 school year. 

b All 1 10 SSCs in the analytic sample. The total number of SSCs in the analytic sample has 
decreased since previous reports because some former SSCs have shut down or changed into 
non-SSC schools over the years. 

C A11 New York City general education high schools that appeared in the 2010-2011 New York 
City Department of Education (NYCDOE) Progress Report. 

d Students are classified as “overage for eighth grade” if they were 14 years old or older on 
September 1 of their eighth-grade school year. 

Calculated by dividing the number of students who took the New York State eighth-grade 
English Language Arts (ELA) exam by the number of ninth-grade students at an ELA 
proficiency level of 1 or 2, at a given school. The results were then averaged across all schools in 
the sample. 

Calculated by dividing the number of students who took the New York State eighth-grade 
math exam by the number of ninth-grade students at a math proficiency level of 1 or 2 at a given 
school. The results were then averaged across all schools in the sample. 

g Teacher turnover rate is defined as the number of teachers who taught in a school one given 
school year but not the next, divided by the total number of teachers in the first of those two 
school years, expressed as a percentage. 

Calculated by taking the average of the average tenth-grade class sizes for math, English, 
science, history, and social studies, all of which were reported in the 2010-2011 New York State 
Report Card as separate measures. 
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2008 (the period of SSC creation considered in this report) and are still in existence. 2 To provide 
additional context, it compares the two groups of SSCs to all 320 other high schools operated 
by the New York City Department of Education (NYCDOE). 

Findings in Table D.l are based on publicly available data for the 2010-2011 school 
year obtained from New York City and New York State school accountability reports. The first 
column lists average values of student, teacher, and school characteristics plus selected perfor- 
mance indicators for the fieldwork SSCs. The second lists this infonnation for all SSCs in the 
study population, and the third presents this infonnation for all other NYCDOE high schools. 
The latter group is intended to provide a broad comparison for SSCs, not necessarily to repre- 
sent the exact mix of schools attended by SSC control group members. 

Table D. 1 lists student characteristics, teacher characteristics, and school characteristics. 
Consider first the student characteristics. As can be seen, the fieldwork SSCs serve a student 
body overwhelmingly made up of low-income, disadvantaged students of color. Students at the 
average fieldwork SSC are similar to those at the average SSC in the study population, but are 
appreciably more disadvantaged than those at other NYCDOE high schools. 

In terms of teacher characteristics, the average fieldwork SSC is similar to the average 
SSC and to other NYCDOE high schools, although specific characteristics vary across the 
groups. Two points are worth noting. First is that a smaller proportion of teachers at the average 
SSC in the fieldwork sample and the study population have a master’s degree plus a certificate 
than is the case for other NYCDOE high schools. Second is that annual teacher turnover 
exceeds 20 percent for all three groups, and is slightly higher at SSCs than at other high schools. 
This high rate of turnover can be especially disruptive for a new small school that is trying to 
create a rigorous, personalized environment. 

In temis of school characteristics, fieldwork SSCs are generally similar to all SSCs, but 
both are quite different from other NYCDOE high schools. For example: SSCs enroll about 400 
students versus 891 students for other NYCDOE high schools; SSCs are concentrated mainly in 
the Bronx and Brooklyn (where the large, failing high schools were located) whereas other high 
schools are more broadly distributed across New York City boroughs; almost all SSCs share a 
campus with another high school (usually on the campus of a high school that was closed); and 
all but 5.6 percent of the SSCs were created during the past eight years, whereas 75 percent of 
other high schools were created before that time. The only similarity in the table between SSCs 
and other NYCDOE high schools is their average class size of roughly 27 to 28 students. 


2 Of the 123 SSCs created during this period, 13 were either closed or converted into a non-SSC-type school, 
such as a middle/high school. 
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Fieldwork Respondents 

Table D.2 describes the backgrounds of SSC principals who were interviewed and Table D.3 
describes the backgrounds of SSC teachers who participated in focus groups. 

At 23 SSCs the current principal was interviewed and at 2 the current assistant principal 
was interviewed. In all cases a senior leadership perspective was represented. In addition, 16 of 
the 25 interview respondents joined their current SSC in its first year, giving them direct 
experience with both the creation and operation of an effective SSC. 3 Furthermore, 21 of the 25 
interview respondents joined their current SSC in its fourth year or earlier, before the SSC 
graduated its first class and typically while it was evolving (a year at a time) from teaching ninth 
grade only to teaching grades nine through twelve. Lastly, 12 of the interview respondents had 
previous experience on the campus where their current SSC is located. 

At 18 of the 25 fieldwork SSCs, focus groups contained three to five teachers. The Re- 
search Alliance fried to recruit focus groups of this size: small enough to provide each partici- 
pant with adequate opportunities to express her opinions and large enough to reflect multiple 
perspectives. To learn as much as possible about both the creation and operation of effective 
SSCs, Research Alliance researchers tried to recruit teachers who had been at their current SSC 
for as long as possible, ideally since its inception. Thus 14 of the 25 focus groups included a 
founding teacher. In addition, focus group participants were typically teachers who had been at 
their current SSC for long enough to have experienced most of its history. The average teacher 
in 23 of the 25 focus groups had been at his current SSC for four years or longer. 4 


Fieldwork Protocols, Coding, and Caveats 

Interviews with SSC principals were designed to take about an hour and focus groups with SSC 
teachers were designed to take about 45 minutes. Attached to this appendix are copies of the 
interview and focus-group protocols that the Research Alliance created and used. These 
protocols focused mainly on two research questions: 

1. What features of SSCs do the professionals who created and operate them 
think are most responsible for their effectiveness? 

2. What do these professionals think were the most serious challenges confront- 
ed while creating effective SSCs? 


3 Some of these respondents originally joined their current SSC as principals, others as assistant principals, 
and yet others as teachers. 

4 Less than 5 percent of the focus group participants were not full-time SSC teachers. Those that were not 
served other roles at the school, for example as guidance counselors or social workers. 
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New York City Small Schools of Choice 
Table D.2 

Principals in the SSC Interviews 


Borough 

Bronx 


Brooklyn 


Manhattan 

Queens 


School 

Pseudonym 

Leadership 
East 
West 
Motivation 
Carroll 
Kentford 
Community 
Bayview 
Riverside 
Rockford 
Division 
Better Way 
Delta 
Fleetwood 

Parkway 
Constitution 
Milford 
Channel Bay 
Mill Creek 
Springside 
Brookside 

Central 

Memorial 

Valley 


Interview 

Respondent 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 

Principal 
Principal 
Principal 
Principal 
Asst. Principal 
Principal 
Principal 

Principal 

Principal 

Principal 


Plainview Asst. Principal 


Y ear of SSC Operation in Respondent Had 

Which Respondent Previous Experience 
Joined It on Campus 


1 

1 

1 

5 

1 

2 

4 

1 

3 

1 

1 

1 

2 

1 


No 

Yes 

No 

No 

No 

No 

No 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

Yes 


1 

5 

5 
1 

6 
1 
4 


Yes 

No 

No 

Yes 

No 

No 

No 


1 Yes 

1 Yes 

1 Yes 


1 


No 


SOURCE: Documentation provided by the Research Alliance for New York City Schools at New 
York University. 


To address the first question, interview and focus group protocols were structured to 
elicit responses both about the core principles of SSCs (rigor, relevance, and relationships) 
and about other factors that are generally considered by the education research literature and 
educational professionals to produce effective schools (for example, supportive leadership, a 
strong professional culture, collective responsibility, and teacher influence over school 
decisions). To minimize the extent to which responses could be steered by interviewers, the 
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New York City Small Schools of Choice 
Table D.3 


Teachers in the SSC Focus Groups 


Borough 

School 

Pseudonym 

Focus Group 
Participants 

Participants’ 

Focus Group Included Average Years at 
F ounding T eacher( s) C urrent SSC 

Bronx 

Leadership 

4 

Yes 

7 


East 

3 

Yes 

6 


West 

3 

Yes 

7 


Motivation 

3 

No 

2 


Carroll 

3 

Yes 

6 


Kentford 

3 

No 

5 


Community 

3 

No 

3 


Bayview 

4 

No 

5 


Riverside 

3 

No 

5 


Rockford 

2 

No 

6 


Division 

5 

Yes 

6 


Better Way 

5 

Yes 

4 


Delta 

3 


5 


Fleetwood 

3 

Yes 

4 

Brooklyn 

Parkway 

2 


4 


Constitution 

2 

Yes 

7 


Milford 

3 

Yes 

7 


Channel Bay 

1 

Yes 

8 


Mill Creek 

2 

No 

7 


Springside 

3 

No 

6 


Brookside 

2 

No 

6 

Manhattan 

Central 

4 

Yes 

5 


Memorial 

3 

Yes 

6 


Valley 

4 

Yes 

5 

Queens 

Plainview 

3 

Yes 

4 


SOURCE: Documentation provided by the Research Alliance for New York City 
Schools at New York University. 


protocols began with questions that were broad and open-ended and continued with questions 
that were more structured and focused. A similar strategy was used to address the second 
research question. 
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The Research Alliance team created codes using the interview protocols, research ques- 
tions, and literature reviews. The process of creating codes was iterative: researchers reviewed 
and edited many drafts of code lists to ensure that codes were mutually exclusive, accurately 
reflected the nature of the interview questions, and served the intent and goals of the research 
questions. In order to check that this was a reliable coding system, four researchers coded the 
same subset of transcripts to discuss discrepancies and missing codes. After two rounds of this 
exercise, the resulting codes were entered into ATLAS. ti (software for qualitative data analysis 
and research) and researchers divided the 25 interview and 25 focus group transcriptions among 
themselves and coded them. After the first round of coding in ATLAS. ti, they examined the 
existing coding scheme again for variations, gaps, and redundancies, and then refined the list of 
codes even further. 

Two caveats are in order about the findings from this fieldwork. The first involves the 
inherent nature of findings from interviews and focus groups. On the one hand, these approach- 
es can make it possible to leam about respondents’ perceptions from their own words. This 
provides a richness of detail that cannot be obtained from more structured approaches. On the 
other hand, this richness of detail calls for researchers to summarize disparate findings, which 
requires subjective judgment. Different researchers can draw different conclusions from the 
same data. Although the Research Alliance took measures to reduce the influence of this 
subjectivity it cannot be removed altogether. A second important caveat is that infonnation from 
SSC interviews and focus groups only represents respondents’ opinions about the factors that 
make SSCs effective. This information can only suggest hypotheses; it cannot confirm them. 
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School Administrator Interview Protocol 
(Research Alliance for New York City Schools) 

Statement to Subjects 

A recent study by an independent research organization, MDRC, found that New York City ’s 
small high schools that were created since 2002 were more effective than other high schools at 
increasing students’ achievement and attainment. The Research Alliance and MDRC are 
collaborating on an extension of this study that seeks to learn why small high schools are 
effective. 

We ’re speaking with you because your school has had a positive impact on student outcomes in 
recent years. Thus, your perspective on how this school operates is critical for helping us 
understand this school’s success. This interview will take approximately 60 minutes and will be 
audio recorded. The questions will focus on your impressions of various aspects of your school, 
the areas in which you think your school is succeeding or struggling, the challenges that the 
school has faced, and where you think the school is headed in the future. 

The Research Alliance will use the information you provide for research purposes only. We will 
keep your responses confidential and will not attribute any comments to any specific individuals 
or schools. Your participation in this study is voluntary. You may refuse to participate or 
withdraw at any time. You may also skip or not answer any questions you prefer not to answer. 

Do you have any questions? 

Background 

To get started, I’d like to ask you a few questions about your background and about how and 
why you became the principal of this school... 

1 . First, how long have you been the principal at this school? 

• Were you involved with the creation of the school? 

2. What were you doing before your current role as principal of this school? 

• Prior principal experience? If so, where? 

3. Please tell me a bit about how you became the principal of this school? 

• Motivation: why this school? 

• What factors influenced your decision to come to this school? 

o Did the school’s being new, small, or theme-based influence your decision at 
all? If so, how? Please describe. . . 
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Ask the following questions if principal was present when school first opened. . . 

You mentioned that you were you involved with the planning of this school prior to its enrolling 
students... I’d like to ask you a bit more about that... 

4. First, please describe how you were involved with the start-up of this school. . . 

• How did this school secure its location and building? 

• Were any key decisions made during this start-up period that have had a profound 
impact on the school’s success? Please describe. 

• What were the major challenges the school faced during this period? 

o Whether, how, by whom, and to what end were these challenges addressed? 

Overall Impressions of the school 

Later in the interview, I'm going to be asking you specific questions about different features of 
this school, but before that, I have some questions about what makes this school effective. 

5. First, does anything set this school apart from other public high schools in New York City? 

• From other small high schools? (Probe for details and examples) 

• Probe for comparisons to other schools that principal is familiar with. 

How does this school measure its success? 

6. Of all of the things that contribute to making a school work, what are the 2 or 3 factors you 
think are most responsible for the successes that you just described? 

• Probe for examples if not offered: how do these factors influence the school’s suc- 
cess? 

• Is there evidence that these are the most important factors or is this a hunch?, e.g., 
teachers tell me this matters 

• Have these always been the most important factors or are they recent develop- 
ments? 

o Other important factors in the past? 

• Probe for process and steps: I’m interested in understanding the process that led to 
these factors contributing to the school’s success... 

o What had to happen? What decisions were made? What made this possible? 
Was this difficult? 

Human Resources 

I’d like to askyou some questions about your teachers and your HR policies... 

7. Please describe the hiring process at this school. 

• How do you recruit teachers? 

• Who screens applications? Who makes decisions about which teachers to hire? 
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• What are you looking for when you hire a teacher to work at this school? 

• What kind of teachers seem to do well in this school? 

• Does the union influence this process? 

8. How are teachers evaluated? How are decisions about terminating teachers or not renewing 
teachers’ contracts made at this school? 

9. Please describe the teachers in this school. . . 

Listen for... 

o Mix of veterans and novice teachers? 
o Primary strengths of the teaching staff? 
o Areas for improvement? 

o Probe for: role teacher characteristics play in contributing to the school’s suc- 
cess? 

Probe: Is there anything about the way the teachers in this school work together that 
plays a critical role in this school’s success? 
o Time in their schedules to work together? 
o School nonns that promote teachers working together? 

Learning Environment 

Let ’s tarn our attention to the learning environment in this school. 

10. Are there aspects of the learning enviromnent that have a big impact, positive or negative, 
on the school’s success? 

Listen for: Safety/discipline; Relationship between staff and students; Academic expecta- 
tions; Student engagement 

• How important are these characteristics relative to those already described? 

1 1 . What is the theme of this school? How, if at all, does the theme influence the way this 
school operates? 

• Influence community partnerships? Curriculum? 

Intermediaries/Community Partnerships 

12. Does the school have any other relationships, past or present, with external partners that 
play a big role in the school’s success? 

• Which ones? 

o Networks, intermediary organizations, community-based organizations, oth- 
ers? 

• What role have these organizations played? 

13. Does this school share the same building with other small high schools? 
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• If so: How, if at all, does the presence of this school(s) in the same building affect 
your school’s success? 

Challenges 

14. What are the biggest challenges facing this school today? 

• Looking ahead, what major challenges, if any, do you anticipate this school facing 
in the future? 

• How do you think these challenges will be addressed? 

Follow-up on impressions of the school 

Earlier in the interview, I asked you about your impressions of the school and about the factors 
that you think are responsible for this school’s relative success... 

15. Has talking about specific aspects of this school prompted any other thoughts about the 
factors most responsible for this school’s success? 

• How important are these relative to the factors mentioned earlier? 

16. In closing, is there anything else that I should have asked you about your school, or that 
you’d like to share, to help us understand this small school operates and why, as a group, 
small schools have been successful? 
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Teacher Focus Group Protocol 
(Research Alliance for New York City Schools) 


Interview code (e.g., WM01): 

School: 

Date: 

Interviewer: 

Interviewees: 


Statement to Subjects 

A recent study by an independent research organization, MDRC, found that New York City ’s 
small high schools that were created since 2002 were more effective than other high schools at 
increasing students ’ achievement and attainment. The Research Alliance and MDRC are 
collaborating on an extension of this study that seeks to learn why small high schools are 
effective. 

We ’re speaking with you because your school has had a positive impact on student outcomes in 
recent years. Tims, your perspective on how this school operates is critical for helping us 
understand this school ’s success. This focus group will take approximately 45 minutes and will 
be audio recorded. The questions will focus on your impressions of various aspects of this 
school and what makes this school successful. 

The Research Alliance will use the information you provide for research purposes only. We will 
keep your responses confidential and will not attribute any comments to any specific individuals 
or schools. We also ask that you not share the responses of your colleagues with anyone outside 
this room. Your participation in this study is voluntary. You may refuse to participate or 
withdraw at any time. You may also skip or not answer any questions you prefer not to answer. 

Do you have any questions before we begin? 

Motivation, Start-up and Professional Culture 

To begin, I’d like to ask you each to describe how you came to this school and what it’s 
like to teach here... 

1 . First, please tell me each of your grades, subjects, and how long have you been teaching at 
this school? 
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• Were any of you full-time teachers prior to coming to this school? If so, where and 
for how long? 

2. Next, please tell me a bit about how you came to teach at this school? 

• What factors influenced your decision to come to this school? 

o Did the school’s being new, small, or theme-based influence your decision at 
all? If so, how? Please describe. . . 

3. Were any of you involved with the planning of the school prior to its enrolling students? 

• If yes, please briefly describe how you were involved. . . 

• Please tell me about any major challenges the school faced during this start-up peri- 
od. . . 

o Were these challenges addressed? 
o If so, how and by whom? 
o How successfully were these efforts? 

4. I'm interested in learning what it’s like to teach at this school... To begin, please describe 
how, if at all, teachers work together at this school. . . 

o When and in what capacity? 

o Fonnal or informal collaborations? Time set aside in the schedule for teachers 
to collaborate? 

o Focus: instructional, non-academic (e.g. students’ social wellbeing), or both 
o School nonns re: autonomy vs. collaboration. Are classrooms open? Do 
teachers regularly observe and critique each others’ work? 


Factors Responsible for School’s Success 

As I mentioned at the beginning of the interview, our collaborators at MDRC have identified 
this school as a school that has been relatively effective. This does not necessarily mean that the 
school is succeeding in every regard (though that may be the case), but rather that the school 
has been successful overall in recent years. Next, I want to askyou to share your thoughts about 
what makes this school successful at preparing and graduating students. 


5. Of all of the things that contribute to making a school work, what 2 or 3 factors do you 
think are most responsible for this school’s success? 

o Probe for examples about how these factors influence the school’s success, 
o Have these always been the most important factors or are they recent develop- 
ments? 

Put bulleted table tent on the interview table so participants can see it,.. 
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This next question has several parts, so we ’ve created this bulleted table tent to help you keep 
track of the components of the question... When new small high schools were being created, the 
theory behind why they would be effective is that they would provide: 1) personalized learning 
environments that enabled close relationships between teachers and students; 2) they would 
emphasize high academic expectations for all students; 3) they would have a curriculum 
focused on a particular theme and that provided students with a real-world learning opportuni- 
ties; and 4) they would have partnerships with external organizations to support the school ’s 
objectives and enhance teachers ’ and students ’ development... 


6. How does your school compare to this general description of small high schools? 

o Are there any notable differences between your schools and this hypothetical de- 
scription? 

o Is one or several of these characteristics more important than the others? 
o How important are these factors at influencing the school’s success relative to the 
factors you identified early (e.g., in response to Question 5)? 


Challenges and Personal Plans 

7. What are the current challenges that this school is facing today? 

o What challenges, if any, do you foresee this school confronting in the future? 
o How do you think these current and future challenges will be addressed? 

Note: May need to focus teachers on challenges unique to this school, as opposed to 
challenges facing all schools. I.E., may need to quickly clarify and refocuses if teachers 
start identifying system- or profession- wide challenges. . . 


Follow-up on impressions of the school 

Earlier in the interview, I asked you about your impressions of the school and about the factors 

that you think are responsible for this school’s relative success... 

8. Has talking about specific aspects of this school prompted any other thoughts about the 
factors most responsible for this school’s success? 

o If participants provide additional thoughts: How important are these relative to 
the factors mentioned earlier? 

9. In closing, is there anything else that I should have asked you about your school, or that 
you’d like to share, to help us understand how this small school operates and why it has 
been successful? 

Thank you very much for your time. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school refonns, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



